ARGOMENTO:

Interessante articolo sui progressi del progetto MCM1
Se mi viene il ghiribizzo lo traduco in italiano....

Alcune novità della migrazione del progetto verso il nuovo cloud

September 8, 2025
Over the weekend we were able to restore the DB2 databases for the website and forums.
It was a redirected restore that first required a fully containerized instance of DB2 running the same OS as we were in Graham cloud, and we ran into issues attempting the restore of the final backups. Both databases are now successfully restored, and we have moved on to containerizing Websphere and IBM MQ.
We were able to restore the BOINC database.
As part of our work on MAM1 we developed an integration testing environment and containerized the BOINC database.
We also did this for the BOINC server components (scheduler, upload and download servers with file_upload_handler, transitioner/validators/assimilators etc.).
Once we get IBM MQ and Websphere up, we will be able to bring the entire system online shortly afterwards.

Sembra che le cose stiano tornando online pian pianino.

boboviz ha scritto:

Sembra che le cose stiano tornando online pian pianino.

September 9, 2025

We are finalizing IBM MQ <-> DB2 <-> BOINC db <-> website axis, which will allow us to bring up the website. If all goes to plan now - we should have the website up tonight.
Once that is solved - we will go through the BOINC stack to ensure nothing catastrophic will happen when once we let traffic through to the scheduler, upload/download servers. Then we can finally start letting the BOINC daemons manipulate state in the BOINC db.

Dai, dai

Messaggio appena ricevuto da WCG sul Boinc Manager:

World Community Grid: We miss you!
We see that you haven't contributed to World Community Grid lately, and we need your help! We encourage you to start participating again today, so download the software and begin contributing again.

Io contribuirei più che volentieri, se solo inviassero WU!
E' arrivato qualcosa ieri, non so cosa o quante wu. Vedo solo i crediti che sono saliti. Poi il buio di nuovo...

Adesso sembra ci sia una buona coda di MCM
sono unità che durano circa 2,5 minuti sul mio AMD 5950
C'è ancora qualcosa che non va: il mio pc PC WIndows 11 viene riconosciuto come "Alpine Linux" chicazz'è

Finally....
October 21, 2025
Finally stress testing rather than correctness testing.
Sent a batch of 100,000 workunits (fast running, not full size in case something crashed.
Thank you for your patience and continued support.

ora le durate sono più regolari, siamo intorno a 75 minuti per WU

Dalla ripresa dell'invio di WU il sito WCG non aggiorna le statistiche di Boincstat.
Qualche idea in merito ?
Grazie in anticipo
K.

Probably has to do with the fact that they are still testing. Latest Update:

October 24, 2025
We have paused uploads and release of test batches while we work on the validation throughput issue.
We think we have identified the root cause of the low validation rate for the MCM1 test batches, and we will send a few more test batches to confirm the fix does works.
If this is the final fix and we see the expected validation rate for pairs of MCM1 uploads for new batches, we will replay the Kafka consumer on the upload events fired for test batches received earlier in the week, and this should idempotently allow the new batch assimilator to process those validations and assign credit.
If the above goes well, we will schedule regular MCM1 batches to resume instead of the test batches.
As volulnteers have noted, we have not yet reconciled uploads of regular MCM1 results submitted before we began sending test batches, and before the migration, but we have those files and will be able to do this in a batch update once the path for new workunits is working as described above.
Naturally, we will resume ARP and MAM only after these issues are fully resolved.

Thank you very much Entity have a good day !
K.

Ancora problemi con la validazione delle wus....

A fronte dei ripetuti problemi (anche oggi è impossibile fare l'upload delle wus), ci sono però delle novità interessanti:

I will quickly chime in to say we have been working on the GPU build for Linux and for Windows for Mapping Arthritis Markers. When that build is ready for testing in beta, we would start distributing batches of workunits within new "MAMG" and "MCMG" beta testing projects. Internally, the name for the application that runs the MAM1 workload is "Mapping Disease Markers on GPUs", or MDMG. That is why the acronym MDMG has come up in some of our updates recently.

Stanno lavorando ad una versione per gpu dell'app. Vediamo cosa ne esce...

Speriamo non siano come il caso di openpandemics dove riuscire ad avere una WU di GPU era esasperante (bisognava martellare robustamente con richieste per riuscire a ricerverne)

A proposito qualcuno è riuscito ancora ad averne qualcuna ???

zioriga ha scritto:

Speriamo non siano come il caso di openpandemics dove riuscire ad avere una WU di GPU era esasperante (bisognava martellare robustamente con richieste per riuscire a ricerverne)

A proposito qualcuno è riuscito ancora ad averne qualcuna ???

Io no, c'è da dire che lascio a lui la scelta e vedo solamente Mapping cancer markers (oltre 20 anni di tempo/elaborazione) da un bel pezzo

Qualcuno è riuscito a prendere qualche WU (CPU) di MAM (Mapping Arthritys Marker) ??

I thought it was still in beta testing. I haven't seen any additional beta work units in several months. From the January 30 update from Jurisica Lab:

Phase 2/3 of MCM1 Validation Backlog Begins - last weekend into this week, we were able to successfully clear a few million more workunits from the MCM1 validation backlog without shutting down the BOINC database. Today, we have started the next phase, focusing on a different category of missed validations. As we run these validators on pre-compiled lists of result file pairs in parallel with the hot path (hot path uses the Kafka topics, firing result upload events so validator_assimilator daemons can batch up transactions of likely reported result pairs and try these "eligible" workunits efficiently), we have one more phase, Phase 3, planned for when Phase 2 completes. When Phase 2 and Phase 3 complete, that should "bring us current" to January 26th (most of the indexing was done on or before January 26th), and we can replay from the Kafka queues through the phase 1, phase 2, and phase 3 cleanup pipelines to get the rest from January 26th to the given day in February (should be a small fraction). The final "cleanup" phase will be re-evaluating workunits marked INVALID by each of the previous Phases 1-3, to see if recovery as VALID of some workunits with yet-more-relaxed criteria makes sense, especially given the large number of hr_class reset workunits we've been running to get around the host os_name and os_version issues from newer BOINC clients.

We may still shutdown the BOINC database temporarily for MCM1 backlog validations - if we choose to do this to complete reconciliation, or to rollout the updated BOINC daemons based on the most recent server release 1.6.1, or go live with our distributed Postgres architecture to replace the current SQLite-per-partition architecture with a more robust, parallelizable backend, we will provide as much notice as possible. We will continue to reiterate this possibility while we are still considering it, as most services will not work if we take the database offline temporarily.

BOINC db_dump for stats will resume once Phase 2 and Phase 3 backlog completes - we have not wanted to put out the stats, because we have not finished reconciling them from all the results that volunteers have uploaded, and we have not felt comfortable deleting results from the result table or result files from the caches or filesystems. We now know we can often credit retroactively, and once we are confident in our reconciliation of MCM1 backlog we will consider the pipeline and architecture stable, and resume db_dump stats dumps as before.

We are pursuing upgrade to BOINC server release 1.6, which has the potential to fix the host os_version corruption issue (see github.com/BOINC/boi...eleases/tag/server_release%2F1.6%2F1.6.1) - we have seen entire error messages from the BOINC client end up in the os_name and os_version fields in addition to the most common case of a truncated single, first letter of the OS that should be recorded there. For example, "W" for Windows. While we cannot guarantee that this release will resolve the issue, and updating to it is still being assessed for feasibility and depends on other large upgrades, we are in a position to finally evaluate and attempt to switch off our legacy BOINC build to track upstream BOINC, and simply patch in the features required to maintain legacy continuity.

MAM1, ARP1, MCM2 releases pushed back - the MCM1 validation backlog issues have stymied many of the cleanup processes required to keep adequate resources available to run multiple projects, especially with large upload sizes and local compute requirements like ARP1. We are in a position to release MAM1 workunits at any time. We have some scientific work to do before we begin distributing "MCM2" workunits using the same application MAM1 uses, as MCM1 batches. Once we are up to date with MCM1 backlog, and begin releasing stats, we will be ramping up other projects and begin approaching the frontend backlog.

OK aspettiamo.
Dopo questo "romanzo"

Almeno adesso abbiamo le idee più chiare

Da www.cs.toronto.edu/~juris/jlab/wcg.html

March 6, 2026

Upcoming Database Migration Will Require Taking BOINC Database Offline Between March 10th-11th - we have been testing the distributed Postgres architecture in our new staging environment. We intend to switch over to our Postgres build of the BOINC server v1.6.1 release next week, between March 10th-11th. The cutover will require a brief pause while we deploy the updated scheduler and feeder to each Postgres worker node, and rollout CDC via Kafka in production to match the staging deployment. With CDC, we will keep the legacy databases in sync with workunit state managed by the Postgres cluster.
CRUD operations from create_work and validator_assimilator daemons will be pipelined by a journaling process to keep workunit state eventually consistent in the legacy BOINC database, so that the current API layer and business logic will continue to work as usual. What we gain is the IOPS available to the BOINC database will scale to the number of servers we deploy in the Postgres cluster, and batch updates to the database will preserve data locality, completing the horizontally scalable design.
This should resolve the long-standing validation issues, high disk utilization for the current BOINC database, and enable us to return to strict homogenous redundancy classes for MCM1 workunits, solve the os_name and os_version buffer overflow and trunctation bug, and give us access to the new features implemented in the more recent BOINC releases, such as running containers as applications on the grid.
We are able to produce and run MCM1 and MAM1 workunits through this system in staging, and will launch ARP1 shortly after we confirm it delivers as expected and described above. We also should then be able to release a larger quantity of workunits than we were previously able to for ARP1 and accelerate the completion of the project.