Changes & To Do¶
Changes Tracker¶
While changes are documented for individual point versions we actually do not push releases out for all of them; they serve more as internal development milestones.
2024-11-22 (v2.0.13)¶
- introducing the option to set
DATABASE_NAMES
through an environmental parameter, to allow setups w/ external databases (thanks to Matt Baker!) - wrapped the filter classes in the
filter_definitions
file into a$defs
root parameter to allow for more structured definition of the filter parameters (e.g.description
etc.) - wrapped the argument definitions into a
$defs
root parameter - added a
local/filter_definitions.yaml
file to allow local overrides of the filter definitions - abandoned the
frequencymaps
collection in favour of thefrequencymap
s inside thecollations
collection- ¡¡¡ This requires re-generation of the maps !!!
2024-11-14 (v2.0.12)¶
- bug fix: the global ENV was previously set to "local" if not called
over HTTP (
ENV = environ.get('HTTP_HOST', "local")
) which led to a wrong "local" context identification when testing from "localhost"; changed toENV = environ.get('HTTP_HOST', "___shell___")
and appropriate tests against this value
2024-11-14 (v2.0.11)¶
- properly have response code before document type in http header ...
- slight modification for parameter processing in
serices/genespans/
- fixed missed splitting of
camelCase
list parameters, introduced some time last week...
2024-11-13 (v2.0.10)¶
- fixed some intermediary issue with a few service calls
- more explicit definition of
array
columns in import tables - documentation now with generated and inlined information from some schemas into bycon.progenetix.org/API/#api-beacon-responses
2024-11-10 (v2.0.9)¶
Mostly work on schemas and generated documentation:
- removing examples etc. from the schemas since they're not tracked and don't represent anything bycon specific
- starting to improve top-level "description"s in schemas, which are then parsed using
markdowner.py
- switching schema model internal references to own server instead of the hardcoded Github repo from the standard schema; e.g.
2024-11-09 (v2.0.8)¶
- some library re-shuffling/condensation
- more work on circle plotting, e.g. separate class
ByconCircleTools
- more documentation
2024-11-07 (v2.0.7)¶
- fixed
/datasets
response, recently broken during class refactoring - added Beacon response documentation from parsing
entity_defaults
with themarkdowner.py
script into/docs/generated
2024-11-07 (v2.0.6)¶
histocircleplot
: /services/collationplots/pgx:icdom-85003?plotType=histocircleplot&plotPars=plotTitle=Genomes+are+circular,+sometimes...- this works for single collations
- so far no gene label options etc.; just for fun...
- some further modification for the importer... scripts which are now reduced to very basic function callers
2024-10-28 (v2.0.5)¶
This update targets the example database:
housekeepers/recordsSampler.py
now provides a method to create an example database (currently hard coded toexamplez
) based on a query towards a target database e.g. such as./housekeepers/recordsSampler.py -d progenetix --filters "pgx:icdom-81703,pgx:icdom-94003" --filterLogic OR --limit 222
- the project's
rsrc/mongodump/
directory contains now- the resulting
examplez
database dump (database contents might change over time) - a dump of the
_byconServicesDB
to be installed alongside for query support
- the resulting
The documentation has been updated accordingly.
2024-10-22 (v2.0.4)¶
- fixed new
/ids
services bug - fixed new
/publication
services bug - modifications to table definitions
- add
individuals.ethnicity
- added variants table generation for most frequent parameters
- add
2024-10-21 (v2.0.3)¶
- bringing the documentation into
bycon
- seriously using Python heredocs, starting with embedding the main information about their functionality into the service caller functions
- project root
markdowner.py
now parses services (and some preference files) into.md
sources for website generation - data import documentation at bycon.progenetix.org/importers/
- revision of
importTablesGenerator.py
2024-10-17 (v2.0.2)¶
- added a
ontologymapsReplacer.py
service app to (re-) create the_byconServicesDB.ontologymaps
collection- this uses the now extended
OntologyMaps
class
- this uses the now extended
- fixed the
schemas
service (recently broken during path parsing refactoring)
2024-10-10 (v2.0.1)¶
- bug fix for query generation for
/datasets
endpoint wheretestMode=true
pragma wasn't interpolated towards alternative response entities - clean up of location and handling of configuration files, both for the package
and local installs
- renamed
bycon/definitions
back tobycon/config
since no other local configs in the package anymore - removed the previous duplication of local configs since now consistent
hierarchy both for local scripts in the project directory as well as in the
web server context (i.e. the
local
directory is at the same level so local references work) - removed
install.yaml
and point tolocal/local_paths.yaml
during installations so that no double definition of local paths is needed
- renamed
2024-10-10 (v2.0.0 "Taito City")¶
General Summary¶
The 2.0.0 version of bycon
represents a milestone release where a general
updated project structure and code base has been established. The main changes involve:
- re-integration of utility functionality into the project
byconServices
,housekeepers
,importers
,rsrc
have been (re-)integrated frombyconaut
- similarly,
byconServiceLibs
are now part of thebycon
libraries
- re-structuring of the
beaconServer
apps and calling- all general Beacon functions in
bycon
are now invoked from a few classes which correspond to the Beacon response types
- all general Beacon functions in
- (ongoing) deprecation of non-standard parameters and functions
Recent Additions¶
- rewrote path parsing and entity selection
- added
path_id_value_bycon_parameter
to map the REST path{id}
to a parameter name (e.g.biosampleId
orfilters
for some services); this removes the separate processing ofrequest_entity_path_id_value
later on
- added
- moved more service functions into dedicated modules; most of the service "apps"
now just call the response modules/functions
- this might led to a future removal of the separate apps/module files
and adding the calls directly to
services.py
in the near future...
- this might led to a future removal of the separate apps/module files
and adding the calls directly to
2024-09-30 (v 1.9.8)¶
- general refactoring of the calling from the
beacon.py
app- remval of all the dedicated caller modules (e.g.
datasets.py
,filtering_terms.py
...) which are now just called by there response functions
- remval of all the dedicated caller modules (e.g.
- more code streamlining for
byconServices
apps- mostly removal of executable status
- more removal of functions from main modules (e.g.
ontologymaps
now uses a separateOntologyMaps
class in a library module)
2024-09-25 (v 1.9.7)¶
- BUG FIX: CNV statistics accessible through the
progenetix.org/services/cnvstats
endpoint had been broken for a while, delivering wrong CNV coverage values.
2024-09-25 (v 1.9.6)¶
- fixes: some broken apps due to incomplete library referencing after update
2024-09-24 (v 1.9.5)¶
- refactoring of service library loading
- the main
bycon/__init__.py
now pre-loads thebyconServices
libraries
- the main
- renaming of
services
tobyconServices
- for the web server the
byconServices
files are still copied toservices
- for the web server the
2024-09-19 (v.1.9.4)¶
byconaut
=> `bycon refactoring- move of many "mature" utility functions from
byconaut
into thebycon
repository following the previousservices
movehousekeepers
importers
- move of many "mature" utility functions from
- move of the
ontologymaps
andpublications
collections fromprogenetix
to_byconServicesDB
- clearer separation between "Beacon core" and "additional services"
use admin db.runCommand( { renameCollection: 'progenetix.ontologymaps', to : '_byconServicesDB.ontologymaps' } ); db.runCommand( { renameCollection: 'progenetix.publications', to : '_byconServicesDB.publications' } );
- clearer separation between "Beacon core" and "additional services"
2024-09-13 (v.1.9.3)¶
- services moved to bycon from byconaut
2024-09-12 (v.1.9.2)¶
- mostly
byconaut
additions (housekeeping scripts etc. and map plot fix)
2024-08-13 (v.1.9.1)¶
- modifications to the VRS Adjacency based format for fusions
- modifications to the
ISCNdefuser
script inbyconaut
for testing fusions
2024-08-07 (v.1.9.0)¶
- adding the new
Adjunction
variant format based on VRS v2.0pre- query method
variantFusionRequest
(usingmate_...
parameters etc.)
- query method
- update of the
byconaut
variantsImporter etc. - general refactoring of datatable import for fields containing objects
- now allows list of ordered parameters for the concatenated values, as well as type definitions
- for details please look at
datatable_mappings.yaml
- rewrite of the
RefactoredValues
class inbycon_helpers
2024-07-30 (v.1.8.9)¶
- rewrite of query stack as
ByconDatasetResults
class- including fix of
analyses
reporting: Now no aggregation towards biosamples anymore - no "non-aggregate" retrieval methods yet; might be added later
- including fix of
- change of variant id query; now not in separate query type
- multi-variant query responses are now aggregated towards response entity (e.g. all variants have to occurr in same individual or - usually - biosample)
- TODO: check multi-variant queries for multi-analyses responses ...
2024-07-17 (v.1.8.8)¶
- We removed of the server URL rewrites to services for e.g.
output=table
pragmas. One has to use the specific endpoints (that have existed since Ocober 2023) and entities, e.g.:- old:
https://progenetix.org/beacon/individuals/?filters=pgx:cohort-oneKgenomes&datasetIds=progenetix&output=table
- now:
https://progenetix.org/services/sampletable/?datasetIds=progenetix&filters=pgx:cohort-oneKgenomes&response_entity_path_id=individuals
- old:
2024-07-10 (v.1.8.7)¶
- fixed wrong
meta.returned_granularity
- changed the
map
schema in line with todays's discussions an pull request #134 - moved the post-processing of exported variants list to a new
VariantsResponse
class and adjusted some of the optional parameters by editing theByconVariant().vrsVariant()
class/method
2024-07-08 (v.1.8.6)¶
- more work on cleaning up the table formats and
byconaut
tools - fix of
response_entity_path_id=analyses
forservices/sampletables
- some Progenetix data content cleanup (e.g. TNM and stage removal from
info.tnm
) - cleaner standard schema definitions for export/documentation (
genomicVariations
will need more work...)
2024-06-24 (v.1.8.5)¶
- bug fix:
variantMinSize
wasn't evaluated if alsovariantMaxSize
was given byconaut
: change in datastructure of status maps and frequencies- we have abandoned the "max value ..." concepts which weren't used anyway and switched to bin coverage and frequency values for high level changes additionally to the all CNV values
- this allows now a simple overplotting of high level gains and losses over the low level/combined CNV histograms
2024-06-19 (v.1.8.4)¶
- changed the values for individual.sex from PATO to EFO (see the Progenetix data notes)
- changed the default for
includeResultsetResponses
fromALL
toHIT
, in line w/ the Beacon standard definition- now empty resultsets won't appear in the list
- the Progenetix / Beaconplus fron-ends call an
ALL
value...
2024-06-11 (v.1.8.3)¶
- restructuring of the whole entity / path ... interpolation - now
set_entities()
andinitialize_bycon_service()
are part ofbycon
__init__.py
- now evaluates the
includeResultsetResponses
parameters (default:ALL
) so that e.g. onlyHIT
responses can be returned in a network context__acknowledge_HIT
removes allexists: false
resultsets__acknowledge_MISS
removes allexists: true
resultsets
- bug fix: removal of the accidential filtering_terms scopes (service entities)
2024-06-06 (v.1.8.2)¶
- removed server side pagination; this should be implemented by clients based on the numbers from the response summary
- FIX: pagination did not work correctly when requesting pavges > 1 using a handover id (due to a "double pagination" at query and response preparation times)
2024-05-27 (v.1.8.1)¶
- re-organized the
local
config location - fixed the correct entity use for non-standard (services...) in test mode
- changed behaviour: results are now shuffled before storing which leads to different result page contents for the same query when applying pagination (i.e. not always the same first 100 etc. samples are returned)
- more fixes for geographic map output after recent code rewrite
2024-05-05 (v1.8.0)¶
With this update the byc
collection object has been abandoned & removed. Parameters
and definitions are now predefined in the global parameters in /bycon/config.py
and (similar to before) populated from external definition files or (for BYC_PARS
etc.)
from processed commansd line, URL or POST parameters. These changes have been propagated
through both bycon
and byconaut
.
2024-05-03 (v1.7.2)¶
This update moves most definitions and pre-processed parameters from the byc
object to the global BYC
object and eliminates (much of) the "passing byc
around"
necessity. Still needs some additional cleanups.
2024-04-25 (v1.7.1)¶
- bug fix (typo in new
info.var_length
calculation)
2024-04-16 (v1.7.0)¶
This version provides a larger update to the multi-variant query concept. We now provide:
- a proper version for using combined
variantQueryDigests
:- /beacon/datasets/progenetix/biosamples?filters=NCIT:C3058&variantQueryDigests=9:21500001-21975098--21967753-24000000:DEL,17:6000000-7687480--7668422-8000000:DEL will e.g. process 2 bracket requests (for small CDKN2A and TP53 deletions) and intersect the samples having both these changes; the same call works, obviously, also for services: /services/sampleplots?datasetIds=progenetix&filters=NCIT:C3058&variant_query_digests=9:21500001-21975098--21967753-24000000:DEL,17:6000000-7687480--7668422-8000000:DEL
- a new (probably to be renamed
variantMultiPars
parameter which only works for POST requests; i.e. each list entry has to be a proper variant query"query": { "requestParameters": { "variantMultiPars": [ { "assemblyId": "GRCh38", "referenceName": "refseq:NC_000009.12", "start": [21500001, 21975098], "end": [21967753, 22500000], "variantType": "DEL" }, { "assemblyId": "GRCh38", "referenceName": "refseq:NC_000017.11", "start": [6000000, 7687480], "end": [7668422, 8000000], "variantType": "DEL" } ] }, "filters": [{"id": "NCIT:C3058"}] }
Also bycon
now supports the default /beacon/datasets/{id}/g_variants
etc. access
pattern for single dataset data retrieval (the use of the datasetIds
parameter is
still not fully standard supported though it had been documented in the framework
... see Github issue #123).
2024-04-03 (v1.6.5)¶
Multi-variant matches first pass¶
We now implement an experimental version of multi-variant matching, to retrieve samples which show co-occurrence of 2 or more variants. This so far is limited to a few parameters:
- NEW:
variantQueryDigests
in the form of9:9000001-21975098--21967753-24000000:DEL
- several of those can be comma-concatenated
- probably not final name or format
- only bracket matches & ranges working (but will change ...)
- different variant types can be used
geneId
has been morphed to a list parameter (though keeping the standard name)- usual comma-concatenation etc.
- here only a global
variantType
can be provided
- future versions to implement mixed matches etc. (e.g. sequence variant & CNV)
Other ...¶
- fixed paginated handovers missing the
info.content_id
parameter which is used by the progenetix-web etc. front ends - changed the
cytoBands
argument totype: array
, to allow definition of multiple cytogenetic regions, e.g. to indicate fusions or bracket requests- this might be a temporary solution for testing purposes and e.g. replaced
by a future parsing of simple statements like
t(8;14)(q24;q32)
- this might be a temporary solution for testing purposes and e.g. replaced
by a future parsing of simple statements like
- moved some cytoband functionality to the main
bycon
package, frombyconaut
, to allow processing of cytoband requests in the main Beacon service byconaut
was restructured for executables, withhousekeepers
andimporters
directories
2024-03-25 (v1.6.4)¶
- added
scopes
tobeacon/filtering_terms/
- fixed bug where the default splitting of input parameters by comma led to over-splitting
of the "embedded list" values in
plotPars
(e.g. usingplotPars=plotGeneSymbols=MYC,T,TP53
would lead toplotPars=plotGeneSymbols=MYC::T::TP53
... and later errors)- now "non-list" strings w/ internal
,
are just re-joined & a warning is created
- now "non-list" strings w/ internal
2024-03-07 (v1.6.3)¶
- configuration changes:
beacon_defaults
file changed toentity_defaults
since only entities defined in it- paths are now defined within the entity definitions, no separate aliases etc.
- local overrides for the Beacon entity defaults now in
local/instance_definitions.yaml
- for byconaut a separate
services_entity_defaults
file provides the additional services (e.g.sampleplots
... pseudo-entities)
2024-03-07 (v1.6.2)¶
- adding a
__collections_response_remap_cohorts(self, colls=[])
function to reformat the collections response for cohorts from the collations format- TODO: define cohorts as separate entities which are read in during collations generation, with their additional parameters etc.
- fixed the openAPI endpoints for collation responses (datasets & cohorts); are incorrect in current Beacon spec.
- bug fix
byconaut
: matrix export was broken since 1.6.1 - exception capture for wrong form values: string values of "None", "none", "Null",
"null" from GET requests are now converted to logical
None
(i.e. removed)
2024-03-06 (v1.6.1)¶
- bug fix: individuals & phenopackets endpoints were broken in 1.6.0 due to missed clean up in query code
- bug fix
byconaut
: vcf & pgsxseg exports were broken in 1.6.0 due to incomplete clean-up of internal variant mapping - bug fix
byconaut
:/geolocations
queries were broken due to needed setting of authorization / granularity, which was handled byrun_beacon_init_stack
which however wasn't used by this service - global change: removal of
run_beacon_init_stack
and inclusion of its function calls in the ubiquitousinitialize_bycon_service
2024-03-04 (v1.6.0)¶
- simplification of internal parameter processing
bycon
/byconaut
- new method "variantsbedfile"- takes over for the previous bedfile/UCSC variants handover generation (removed
from
handover_generation
) - defaults to bedfile download from variant query
output=ucsclink
creates the UCSC link with added bedfile payload
- takes over for the previous bedfile/UCSC variants handover generation (removed
from
byconaut
- refactoring offrequencymapsCreator.py
to use the standard collation bundle generation instead of custom queriesbyconaut
- change of collation retrieval to work now withcollationTypes
parameter- this allows to e.g. get all clustered CNV plots for a classification tree:
/services/collationplots/?collationTypes=NCIT&minNumber=200
- this allows to e.g. get all clustered CNV plots for a classification tree:
- bug fix: broken server address in handovers
- bug fix:
id
specific query for collations was broken (delivered all) - bug fix: mapping of basic chromosome ids (
9
) to refseqs was broken
2024-02-24 (v1.5.2)¶
- new
instance_definitions.yaml
config document inlocal
- this allows to override Beacon instance parameters based on the URL the service is running under, enabling multiple Beacon instances per server
- new
analysis_operation
inanalyses
(i.e. "pgxAnalysis" in Progenetix database model) allows now the filtering of analyses based on the type of genomic profiling performed with its (so far) values:"analysis_operation.id":"EDAM:operation_3961", "analysis_operation.label":"Copy number variation detection"
"analysis_operation.id":"EDAM:operation_3227", "analysis_operation.label":"Variant Calling"
2024-02-21 (v1.5.1)¶
BYC_PARS
now as a global parameter, not passed around in methods (formerlybyc["form_data"]
)- byconaut: fix of parsing of plot variables (which can be shown through
&showHelp=true
) - hot fix: added "protected" status for
external_references
in general empty field clean-up since the object is required by the front-end (even if empty list)
2024-02-20 (v1.5.0)¶
- refactoring global configs into
bycon/config.py
to slowly get rid of some of thebyc
-> ... imports (e.g. global DB parameters, collecting warnings...) - removal of
service_config
parameter & generator code frombycon
, nof handled explicitely in the different byconaut services - fixed
geneVariantRequest
to be selected as type when ageneId
parameter is provided - fixed handovers for non-default datasets by adding the
datasetIds
parameter to the handover- bug was based on older design retrieving the dataset id directly from the handover in the temp storage ...
- moved (partially so far)
external_references
toreferences
in biosamples- reference objects are now standard
id
,label
term objects references
is an object, i.e. the items are keyed{"pubmed": {"id": "PMID:1234567", ...
- regeneration of the reference structure from Beacon/Phenopackets is done at export time
- reference objects are now standard
byconaut
with new/services/samplemap/
endpoint for plotting geolocations of sample data after standard Beacon queryfilter_definitions
- fix for arrayexpress series processing (now
AEseries
) - changed
collatiionType
PMID
=>pubmed
- fix for arrayexpress series processing (now
- fixed
uploader
fail due to missing import
2024-02-07 (v1.4.2)¶
- more consolidation of argument/cgi parsing libraries
- filter flag parameters now properly defined in
argument_definitions.yaml
- use of a common
db_config
object for database configuration parameters - piecemeal move of placeholder parameters from
config.yaml
(either through removal & fallbacks in methods, or through definition inargument_definitions
) - working on error responses...
- removed generic pre-processing methods for error & general responses
- fixed a bug which allowed non-matching filters to pass
- fixed examples in
tests
- fixed
POST
processing (wrongfilters
nesting as in examples ...) - fixed the OpenAPI "endpoints" info for the entry types
- The
openAPIEndpointsDefinition
parameter in progenetix.org/beacon/map/ should now point to working definitions per entity, e.g. progenetix.org/services/endpoints/biosamples
- The
2024-02-02 (v1.4.1)¶
- new
ChroNames
class for accessing chromosome and refseq ID mappings (still utilisingbyc["refseq_chromosomes"]
as input, read during init fromrsrc/.../refseq_chromosomes.yaml
) byconaut
: Added a newplotType=histosparklines
plotting option. It basically modifiesplot_defaults
parameters for minimal histoplots (no border, no background, small and narrow), e.g. for use in mouse-overs or in tables- one can still override those parameters, e.g. with
&plotPars=plotDendrogramWidth=50::plotAreaHeight=32
- one can still override those parameters, e.g. with
byconaut
: added option to use sequence id values for plotPars.plot_chros, e.g.plotPars=plot_width=980::plotChros=NC_000023.11
- fix: a
g_variants
endpoint w/o any parameter led to a query error - removal of some
byconaut
code remnants - some internal reshuffling; e.g. move of export/print helpers to from
parameter_parsing
tobeacon_response_generation
andbycon_helpers
2024-01-18 (v1.4.0)¶
- fix of
plotType
parameter as separate one (used in byconaut) - fix of wrong parameter mapping for
geo:GSM....
filters- filter definition still pointed to
external_references.id
instead ofanalysis_info.experiment_id
- filter definition still pointed to
byconaut
: move to general use ofbyc["form_data"]
for arguments (i.e. requiring the command line arguments to have been parsed into this object)- fixed the
byconaut
cytomapper
service by addingcyto_bands
andchro_bases
to the list ofvariant_pars
2024-01-10 (v1.3.9)¶
Bugfix release. Some default values provided in argument_definitions.yaml
file were overriding pre-processed values, leading to e.g. to an
endless loop in the handover generation.
2024-01-09 (v1.3.8)¶
- argument parameter redesign
- definition of all parameters/arguments (web & local) in
argument_definitions.yaml
- parameters not defined there will not be processed anymore (however, there are some
placeholders like e.g.
mode
orkey
which are not utilized by standard methods and can be co-opted for custom inputs) - plot parameters are provided as a single string to
plotPars
, with individual parameter pairs concatenated by::
- in GET:
plotPars=plot_chros=8,9,17::labels=8:120000000-123000000:Some+Interesting+Region::plot_gene_symbols=MYCN,TP53,MTAP,CDKN2A,MYC,ERBB2::plot_width=800
- in CMD:
--plotPars "plot_chros=8,9,17::labels=8:120000000-123000000:Some Interesting Region::plot_gene_symbols=MYCN,TP53,MTAP,CDKN2A,MYC,ERBB2::plot_width=800"
- in GET:
- definition of all parameters/arguments (web & local) in
- modification of the
prdbug
helper
2023-12-18 (v1.3.7)¶
- added handling for user specific granularity permissions
- so far
user_name
is just taken from a form parameter and then stored asbyc
root parameter (throughset_user_name
) - local processing (
env
) sets this tolocal
(and has a defaultrecord
) granularity - dataset specific, user specific maximum granularities can be set in
authorizations.yaml
which can be extended / overwritten from settings inlocal/authorizations.yaml
(similar tobeacon_defaults.yaml
etc.) - future updates are planned to handle proper interpretation of
user_name
and proof of authorization...
- so far
- configuration: the basic parameters from
config.yaml
are now stored asbyc
root parameters and not kept in a mix of root &config
2023-11-20 (v1.3.6)¶
- modified
BeaconDataResponse
to keep theresultSetsResponse
structure while remobving theresults
from each set, to allow resuult set specific handover delivery (labeled as CUSTOM) - moved all cytoband library code into
byconaut
(FUTURE considerations for this in case Beacon supports cytobands ...) byconaut
plots now directly use the database saamples format for plotting variants, w/o going through the canonical variant creation (this incurred a huge penalty)- reminder that
byconaut
plots use theplotType
parameter instead ofoutput
byconaut
now has a color code mapping for the different (EFO, DUP/DEL ...) variant types; this allows to assign customplot_dup_color
etc. parameters while keeping the available variant types (variant_state.id
) separated (seebyconaut -> local.plot_defaults
)
2023-11-17 (v1.3.5)¶
- more removal of non-standard components into
byconaut
, e.g. for file generation such as.pgxseg
- adding experimental
target
field to items infiltering_terms
response - adding
aminoacidChange
andgenomicAlleleShortForm
to request parameters (this was a bug fix - they were already activated but not in the.json
version)
2023-10-31 (v1.3.4)¶
This update is mostly addressing the further removal of methods specific for "beyond Beacon" functionality (e.g. variant binning and calculations for CNVs, plotting ...).
2023-10-25 (v1.3.3)¶
Most of the "special outputs" code has been moved to byconaut -> services.
For legacy reasons (e.g. use by pgxRpi) the webserver configuration needed some
rewrites ... They only apply for the Progenetix use case and are not needed if
sticking to the Beacon formats or if following the use of the new apps like
services/vcfvariants
). Our (temporary) mappings are:
RewriteEngine On
# The following rules are for backward compatibilitty with pgxRpi before Oct 2023
RewriteCond %{QUERY_STRING} ^(.*?output=\w*?table.*?)$
RewriteRule "^/beacon/biosamples.*?$" /cgi-bin/bycon/services/sampletable.py?%1&responseEntityId=biosample [PT]
RewriteCond %{QUERY_STRING} ^(.*?output=\w*?table.*?)$
RewriteRule "^/beacon/individuals.*?$" /cgi-bin/bycon/services/sampletable.py?%1&responseEntityId=individual [PT]
RewriteCond %{QUERY_STRING} ^(.*?output=\w*?table.*?)$
RewriteRule "^/beacon/individuals.*?$" /cgi-bin/bycon/services/sampletable.py?%1&responseEntityId=individual [PT]
RewriteCond %{QUERY_STRING} ^(.*?output=\w*?matrix.*?)$
RewriteRule "^/beacon/analyses.*?$" /cgi-bin/bycon/services/samplematrix.py?%1&responseEntityId=analysis [PT]
RewriteCond %{QUERY_STRING} ^(.*?output=vcf.*?)$
RewriteRule "^/beacon/biosamples/([^/]+?)/g_variants.*?$" /cgi-bin/bycon/services/vcfvariants.py?%1&biosampleIds=$1 [PT]
RewriteCond %{QUERY_STRING} ^(.*?output=pgxseg.*?)$
RewriteRule "^/beacon/biosamples/([^/]+?)/g_variants.*?$" /cgi-bin/bycon/services/pgxsegvariants.py?%1&biosampleIds=$1 [PT]
2023-10-20 (v1.3.2)¶
This version removes the complete bycon_plot
code (i.e. moves it to byconaut
).
It still needs the further disentangling of the other alternative response options
(.pgxseg
, .pgxmatrix
...) from the resultsets generation; this soon will follow
blueprint of the plot code removal.
CAVE Now all plotting options have been shifted to the /services/collationplots
and /services/sampleplots
entry points.
2023-10-20 (v1.3.1)¶
This version provides another step in moving "non-standard" Beacon responses tp
the byconeer
project.
- creatiing a
.../services/sampleplots/
entry point which will be used to handle the sample (strips/clustered; histoplots from search results ...) web plotting instead of adding theoutput=histoplot
etyc. option to standard Beacon queries- plot types can now be specified through
plotType=samplesplot
etc.
- plot types can now be specified through
- some class (
ByconResultSets
) restructuring to allow plot outputs (this will be changed further, probably moving the whole plot ... classes and methods tobyconeer
)
2023-10-12 (v1.3.0)¶
This is an extensive internal code update which
- moves service response generation to byconaut (implemented as
ByconServiceResponse
class w/ its methods) - removes many of the methods from
service_utils
since they have been implemented in the beacon or services response classes and (mostly) limits the library to general/initialization methods (still more to clean...) - fixes some inconsistencies (e.g. snake vs. camel cases in paths where sometimes
non-standard versions were documented - now using the Beacon v2 defaults such as
beacon/filtering_terms/
instead ofbeacon/filteringTerms/
) - similar for geo queries (e.g.
geoLatitude
as query parameter instead ofgeolatitude
) though this is "BeaconPlus" anyway
CAVE: These changes also affect the front-ends (progenetix-web
, beaconplus-web
etc.)
which need to be recompiled from the latest versions
2023-08-30 (v1.2.2)¶
- some defaults cleaning (e.g. removal of non-standard paths from built in
beacon_defaults
)
2023-08-25 (v1.2.1)¶
- clean-up of info response m(all entryType schemas shown now)
- modification of entity_defaults format
- use of
beaconCollectionsResponse
for services & deprecation ofProgenetixServiceResponse
mongo_test_mode_query
(needs to be propagated more...)beaconplus
domains support
2023-08-22 (v1.2.0)¶
- fix of
filterLogic
parameter for forced global$or
- more reshuffling of defaults and config parameters
- merged
beacon_mappings
intopbeacon_defaults
- moved
config.yaml
tobycon/config/
- splitting of the
beacon_defaults
parameters into standard parameters, e.g. for the main entry types, into thebeacon_defaults.yaml
file inbycon/config/
, and custom parameters (e.g. Progenetix'phenopackets
entry type definition or some aliases) into the/local/
location
- merged
- streamlining of
__init__.py
andread_specs.py
w/ respect to those changes - concurrent
byconaut
update
2023-08-21 (v1.1.7)¶
This update continues with the disentangling of "package inherent" defaults and definitions and "local" ones. Partcullarly:
- standard Beacon entity definitions arte now part of the package configuration,
i.e.
bycon/config/beacon_defaults.yaml
has now the entities, and additional entities are then provided frombycon/local/beacon_defaults.yaml
(which is copied from the project root/local/beacon_defaults.yaml
) duringupdev.sh
)- examples are the Progenetix specific data in the
info
entity or the non-standardphenopackets
entry type
- examples are the Progenetix specific data in the
- similar for
dataset_definitions
...
2023-08-21 (v1.1.7)¶
This update continues with the disentangling of "package inherent" defaults and definitions and "local" ones. Partcullarly:
- standard Beacon entity definitions arte now part of the package configuration,
i.e.
bycon/config/beacon_defaults.yaml
has now the entities, and additional entities are then provided frombycon/local/beacon_defaults.yaml
(which is copied from the project root/local/beacon_defaults.yaml
) duringupdev.sh
)- examples are the Progenetix specific data in the
info
entity or the non-standardphenopackets
entry type
- examples are the Progenetix specific data in the
- similar for
dataset_definitions
...
2023-08-16 (v1.1.6)¶
- bugfix release for service items
2023-08-16 (v1.1.4 => v1.1.5)¶
- some changes to defaults & mappings parsing
- merging content of "beacon_defaults" & "service_defaults" (if existing) files during init into "beacon_defaults"
- new requirement:
deepmerge
(removedpydeepmerge)
)
- some reshuffling/fixes of entry type defaults
- refined
GeoLocation
schema - now in model...common and referenced there - v1.1.5 was a bugfix immediately after the update ...
2023-08-11 (v1.1.2 -> 1.1.3)¶
- move the new
histoheatplot
method code to use ImageDraw instead of SVG raw for the heat strips (i.e. base64 encoded individual PNG strips)- e.g. reduces size of 9.3MB example to 188kB
- 1.1.3 fixes a combination query bug
2023-08-10 (v1.1.1)¶
- services: new frequency plot type
histoheatplot
:- also some new related parameters, e.g.
plotHeatIntensity
- /services/intervalFrequencies/?datasetIds=progenetix&filters=NCIT:C,!NCIT:C3247,!NCIT:C3510&filterPrecision=start&collationTypes=NCIT&output=histoheatplot&plotAreaHeight=20&plotRegionGapWidth=&plotChros=3,17&plotHeatIntensity=1.5&plotGeneSymbols=TP53,BCL6&plotDendrogramStroke=2
- also some new related parameters, e.g.
intervalFrequencies
now supports start-anchored greedy term matches as well as exclusion of individual terms through!
prefix (as for normal filter searches)
2023-08-09 (v1.1.0)¶
- fixed gene position queries (had hardcoded
progenetix
database ...) - general overhaul of query generation
class ByconQuery
- called directly in
query_execution
- removed some variant query types (e.g. "type only")
- better handling of
id
type queries (including the retrieval of the associated variants) - move to collection of query objects per entity instead of collection (is mostly a logical change for general use of entities instead of the database impolementation as target - that is resolved at the query execution)
- TODO: probably will move from the dynamic variant query type detection to a stacked "test one after the other with hard-coded parameter checks" just for sanity reasons - but right now see above
- TODO: variant requests and request type detection still not part of class
- TODO: geo queries into class ...
- there is now a
force_empty_plot
(forceEmptyPlot) parameter so that sample queries w/o any CNV (e.g. from cancercelllines.org samples) generate an empty strip, to add non-CNV as labels - fixed error in
interval_utils
(renamed config key...) - fixed associated
byconaut
errors
2023-07-26 (v1.0.72)¶
- cleaning of handovers
- no "all variants from matched biosamples" ... anymore due to performance problems
- now variant storage and handovers only for matched variants - i.e. if there was a variant query - or if a sample, individual ... had been requested by path id
- CNV VCF fix
- some general handover creation cosmetics
- VRS schema components moved
2023-07-25 (v1.0.71)¶
- finished the
ByconVariant
vrsVariant
method and implemented this as the format for the Beacon variant response - added sorting to
.pgxseg
files - added compact
VRSallele
andVRScopyNumberChange
schemas & using them inByconVariant().vrsVariant(v)
byconaut
¶
- moved
geoloc_utils
tobyconaut/services/lib
- improved index generation for
2dsphere
indexes - moved
geolocs
to_byconServicesDB
and adjusted code accordingly - fixed
frequencyMapsGenerator
for the new database
2023-07-24 (v1.0.70)¶
- refactored the VCF export into
exportfile_generation
- minor pagination code cleanup
2023-07-24 (v1.0.69)¶
- fixed datatable & pgxseg download error (introduced w. 1.0.68)
- internal change of more consistant use of
genomicVariant
for variant entity, schema
2023-07-23 (v1.0.68)¶
- first production version of
ByconVariant
class and consecutive retirement of...vrsify
functions - new defaults in
variant_type_definitions
per EFO type:dupdel_state_id
snv_state_id
VCF_symbolic_allele
- fixed wrong parameter mapping of
alternate_bases
(introduced ... when?) - docs: split changes & to dos...
2023-07-19 (v1.0.67)¶
- fix
.pgxseg
file loader bug (thanks Huan Zhang!) - starting
variant_mapping.py
for a consolidatedByconVariant
class
2023-07-13 (v1.0.66)¶
- modified
return_filtering_terms_response
to parse over the collations from different datasets- check & streamline
- modified
retrieve_gene_id_coordinates
(some error catching) - introduced
housekeeping_db: _byconHousekeepingDB
forquerybuffer
andbeaconinfo
- accordingly changed
byconaut
, e.g. housekeeping ->beacon_info_coll
- accordingly changed
- changed
services_db: _byconServicesDB
forgenes
2023-07-11 (v1.0.65)¶
- streamlining of schema file parsing
- the schema file root is now hard coded to
path.join( pkg_path, "schemas" )
- schemas are identified by their unique name (
beaconMap.json
) or patent dir / default combination (.../biosamples/defaultSchema.json
) - ... which means those have to be unique
- this removes all the schema path definitions from
config.yaml
- the schema file root is now hard coded to
2023-07-06 (v1.0.64)¶
- fixing camelCase / snake_case errors for
filteringTerms
&genomicVariations
entry types (I hope; those came up in 1.0.62 after streamlining config parsing...)
2023-07-06 (v1.0.63)¶
- cleaning up the install.py script
- fix in byconaut for local preferences when executing from local repo (where this lead to an empty stats database -> beacon errors...)
2023-07-05 (v1.0.62)¶
- bug fix sample plots
- internal function re-organization (
initialize_bycon_queries
deprecated & replaced byparse_filters
&parse_variants
) - also addition to the ENV config in byconaut
2023-07-04 (v1.0.61)¶
- adding
BYCON_MONGO_HOST
environment variable to enable other MongoDB host thanlocalhost
(which remains fallback/default) - thanks @fliem for #17 - added
--noo-sudo
to install.py - thanks @fliem for #19 - more tweaking of configuration reading
2023-06-30 (v1.0.59)¶
- fixed new bug in variant parameter parsing
- fixed wrong parsing of command line list arguments (e.g.
--filters
) install.py
now adds the/local
configurations to the/bycon/beaconServer/local/
directory- execution of command line beacon has access to them (not only apps in the server)
- files are removed by
/updev.sh
before packaghe build
- added some command line examples to
installation.md
2023-06-28 (v1.0.58)¶
- extensive renaming/-shuffling, e.g.:
refseq_chromosomes
now inrsrc/genomes/grch38
(only grch38 so far but this is all we currently use...)- also
parse_refseq_file
and__get_genome_rsrc_path
functions
- also
variant_request_definitions
andvariant_type_definitions
config files fromvariant_definitions
(separating the query config from the type mappings)cytoband_utils
=>genome_utils
set_genome_rsrc_path
wrapper for cytoband and interval functions
- fix for file uploader issues
- TODO: documentation on website & lazy loading (e.g. interpolating
sample
tobiosample_id
; maybe just use column order ...)
- TODO: documentation on website & lazy loading (e.g. interpolating
2023-06-26 (v1.0.57)¶
- more disentangling of configuration between
byconaut/services
and/bycon
2023-06-21 (v1.0.56)¶
info_db
=>services_db
parameter renaming- fix of missing interpolation of query parameters into the response metadata
2023-06-21 (v1.0.55): Removal of /services
¶
- removed the
/services
part from the bycon package - it is now maintained inbyconaut
with the local location being defined ininstall.yaml
- fixed some tests
2023-06-21 (v1.0.54)¶
- adding
normalize_pgx_variant
2023-06-13 (v1.0.53)¶
- now probe plotting as implemented with auto-path detection based on callset values
analysis_info: { experiment_id: 'geo:GSM498847', series_id: 'geo:GSE19949' }
leads to{server_callsets_dir_loc}/GSE19949/GSM498847/{probefile_name}
- example: progenetix.org/beacon/biosamples/pgxbs-kftvkafc/?output=samplesplot&plot_chros=3,5,6,14
2023-06-12 (v1.0.52)¶
- modified
handover_definitions
to follow the specification: handoverType
now as in spec, also using public identifier where possible (so far"id": "EDAM:3016", "label": "VCF"
for all VCF h->0)- since now different handovers can have the same
handoverType.id
this required the addition of aninfo.contentId
value for the frontend to disambiguate - starting the work on the
arrayplot
(?) plot type, including a new method for getting probe file paths (not yet activated) - several field changes in biosamples, to align w/ main Beacon v2 default schema:
sampledTissue
=>sampleOriginDetail
description
=>notes
timeOfCollection.age
=>collectionMoment
2023-06-06 (byconaut)¶
- moved MongoDB index generation to
housekeeping.py
- started to organize actions in
housekeeping.py
w/ user prompts
2023-06-05 (v1.0.51)¶
- fixed pgxseg file reader (broken reference_name ... parsing after recent chromosome fix)
- removed some publication libraries/schemas only used in byconaut
2023-06-02 (v1.0.50)¶
- fixed missing chromosomes in
.pgxseg
exports - age search now with 2 values possible (e.g. to set a range)
2023-05-31 (v1.0.49)¶
- added
days_from_iso8601duration
method - added in individuals
index_disease.onset.age_days
field- populated using the new method w/ a byconaut "housekeeping" script
- added a first
alphanumeric
filter type & parsing - this enables now an age filter query, e.g.
filters=age:<=P35Y2D
2023-05-31 (v1.0.48)¶
- addresses VCF export bugs #14, #15 and #16 (thanks David!)
- fixes some
null
value complaints from the Beacon verifier (thanks Dmitry!)
2023-05-26 (v1.0.47)¶
- changed handover id format from
pgx:handover:biosamples
to CURIE-compatiblepgx:HO.biosamples
etc. style (see Beacon #83)
2023-05-25 (v1.0.46)¶
- changed the internal schema for genomic variants
- simplification, e.g. using everywhere
variantState
and flattening of thelocation
object - adding
MT
chromosome support (though not necessarily for searches etc.)
- simplification, e.g. using everywhere
2023-05-24 (v1.0.45)¶
instantiate_schema
has been rewritten; this lead to a number of code adjustments (e.g. usually starting w/ the schemas themselves, not w/properties
when instantiating) and bug fixes (mostly capturing errors from default "none" values)
2023-05-22 (v1.0.44)¶
- code clean-up
- removal of unnecessary "return" staements
- re-structuring of
argument_definitions.yaml
in preparation for common parsing of input values
2023-05-17 (v1.0.43)¶
- rewrite of
.pgxseg
processing intoByconBundler
class- includes
callsets_variants_bundles
andcallsets_frequencies_bundles
for plot object generation
- includes
- plot labels fix for labels starting at the
0
base
2023-05-12 (v1.0.41)¶
- complete refactoring of plot code as
ByconPlot
class - renamed library
bycon_plot.py
2023-05-09 (v1.0.40)¶
- new
services/samplesPlotter
entry point- currently specific for file uploads (handling of DB calls see below)
- completed the integration of the new
bycon
plotting - new
plotCytoregionLabels
plot labeling parameter
2023-05-03 (v1.0.38)¶
- added method to subset samples for multi-histogram generation using the
groupBy
parameter for matching offilter_definitions
classes - removal of deprecated
cgitb
use & replacing it w/ a simple Exception/traceback wrapper - expansion of the plotting documentation
- move of the plot arguments parsing to
parameter_parsing.py
2023-05-01 (v1.0.37)¶
- more plotting: now histograms and samples - if >3 - get a dendrogram to indicate
the clustering results
- clustering can be suppressed by
&plotClusterResults=false
- clustering can be suppressed by
2023 01-04¶
2023-04-27 (v1.0.36)¶
- plotting & clustering of samples
2023-04-27 (v1.0.35)¶
- new
custer_utils.py
library for cluster matrix generation and clustering- uses
scipy.cluster
- first test implementation for frequency maps; auto-clusters if >2 using the concatenated gain and loss frequencies per standard binning
- tree plotting
- uses
- plot gene label refinement
2023-04-25 (v1.0.35)¶
- more plot_utils goodness - staggered labels
- /services/intervalFrequencies/?plotChros=2,8,9,17&labels=8:120000000-123000000:Some+Interesting+Region&plot_gene_symbols=MYCN,REL,TP53,MTAP,CDKN2A,MYC,ERBB2,CDK1&filters=pgx:icdom-85003&output=histoplot
- /beacon/biosamples/?datasetIds=examplez,progenetix,cellz&referenceName=9&variantType=DEL&start=21500000&start=21975098&end=21967753&end=22500000&filters=NCIT:C3058&output=histoplot&plotGeneSymbols=CDKN2A,MTAP,EGFR,BCL6
2023-04-20 (v1.0.34)¶
- new
geneSymbols
parameter for plot labeling allows to add gene labels to the position of a given gene on a plot- http://progenetix.org/services/intervalFrequencies/?geneSymbols=MYC,ERBB2,CDK1&filters=pgx:icdom-85003&output=histoplot
- TODO: still needs a "no-overlap" shifting method for closely spaced labels (as in the Perl version...)
2023-04-17 (v1.0.33)¶
- refinement of
histoplot
options; now as the standard for standard Beacon results by using the&output=histoplot
pragma - change of
histoplot
handover (id: 'pgx:handover:histoplot'
) to use the plotting option instead of a redirect to the PerlPGX
version
2023-04-14 (v1.0.32)¶
- basic implementation of plot labels for the collation frequency plots, using
the
plot_region_labels
parameter - switch of the cancercelllines.org UI to use this plotter instead of the Perl based PGX one (only for the pre-computed collations)
2023-04-14 (v1.0.30)¶
- added
plot_utils.py
andplot_defaults.yaml
; now there is a first method for plotting histograms- so far limited to CNV histograms of many or few whole chromosomes from pre-computed frequencymaps
- no separate sevice so far; can be invoked from the
intervalFrequencies
service with added&output=histoplots
pragma - so far no marker addition etc.
- progenetix.org/services/intervalFrequencies/?filters=pgx:icdom-85003,pgx:icdom-87003,pgx:icdom-81403&plotChros=7,8,9,13,17&plot_title=CNV+Comparison&output=histoplot&size_plotimage_w_px=800&plot_chro_height=18
2023-04-12 (v1.0.29)¶
- fixed broken dataset selection (bug introduced w/ v1.0.28)
- moved dataset parsing to separate library
lib/dataset_parsing.py
2023-04-11 (v1.0.28)¶
- fixed the filter processing where "correctly looking but not existing" filter patterns were pruned from the query instead of being kept & leading to a mismatch
- added 2 types of warnings for such cases:
- undefined filter pattern
- correct pattern but value not in database
2023-03-31 (v1.0.27)¶
- FIX:
ids
service (and therefore identifiers.org resolver) was broken due to custom & degraded config path
2023-03-31 (v1.0.26)¶
- removal of VRS classes since VRS is about to switch to EFO
- added
EFO:0020073: high-level copy number loss
from the upcoming EFO release - adding
examplez
dataset definition
2023-03-30 (v1.0.25)¶
Bug fix release:
- fixed
output=text
errors - some argument parsing bugs that crept in with last release
- library re-shuffling w/ respect to
byconaut
2023-03-27 (v1.0.24)¶
- added
output=vcf
option for variant export & made it default for thephenopackets
entity- VCF export is basic & hasn't been tested for round trip compatibility
- added filter exclusion flag:
- for POST a Boolean
"excluded": true
- for GET prefixing a term by an exclamation mark (e.g.
!PATO:0020002
) - this is a BeaconPlus feature - see issue #63 in
beacon-v2
- for POST a Boolean
2023-03-02 (v1.0.22)¶
- v1.0.22 fixes the
testMode=True
call
2023-02-05¶
bycon
now available as Pypi packagepip install bycon
2023-01-15¶
- create bycon documentation subdomain & configure Github pages for it
Known Bugs¶
- ... they change all the time ...