Scaling Issues, 30MB Sample transactions, and persistent.list.PersistentList

We have a well populated site currently containing about 20,000 samples and 480,000 analyses. While I would expect our database to be large given the quantity of objects, we are having a major performance issue that I cannot seem to track down.

Every time we perform a write transaction on the site involving a Sample or Analysis (child of a Sample), not only does the action take anywhere from 10 seconds to 10 minutes, we get this warning:

UserWarning: The <class 'persistent.list.PersistentList'>
object you're saving is large. (30290947 bytes.)

Perhaps you're storing media which should be stored in blobs.

Perhaps you're using a non-scalable data structure, such as a
PersistentMapping or PersistentList.

Perhaps you're storing data in objects that aren't persistent at
all. In cases like that, the data is stored in the record of the
containing persistent object.

In any case, storing records this big is probably a bad idea.

If you insist and want to get rid of this warning, use the
large_record_size option of the ZODB.DB constructor (or the
large-record-size option in a configuration file) to specify a larger
size.

Now, most of the sluggishness itself is coming from ConflictErrors which are causing transactions to run multiple times. However, what I am gathering out of this error message is that every single Sample object is currently around 30MB in size, which seems absurdly large given the attributes being stored. :face_with_monocle:

According to the Zope Documentation, the best way to reduce the chance of ConflictErrors is to reduce the size of the transactions, so that there is a smaller window for a conflict to occur. However, I am a bit unsure why these Sample objects are so large in the first place. It was not this sluggish when we first started entering production data about a year ago, so my thought is that each Sample (AnalysisRequest) is somehow getting larger every time a sample is created?

Anyone have any experience with this issue, or have any understanding of how persistent.list.PersistentList would be causing Sample objects to grow so large over time?

Hi @faytrow,

when was the last time you packed your database?

Maybe you are also storing large images e.g. directly pasted into the results interpretation of some samples?

As far as I know FireFox converts such images directly into base64 and this get all stored in the text field (also see Handle inline images in Results Interpretation by ramonski · Pull Request #1344 · senaite/senaite.core · GitHub)

Ramon

2 Likes

Thanks for the reply @ramonski. I read your reply last fall, but never got back with a response since I’ve been doing a lot more operations than development.

The database packing did not solve the issue, and I do not believe we had stored any images in result interpretations (I believe our only images are attachments to Samples, and the reports generated through senaite.impress).

The site was still on senaite.core v2.0.0 and senaite.lims v2.0.0 at the time, so I did a full ETL pull of all the business data into a fresh v2.4.0 instance. So far I have not seen the persistence issue, and it has made a 2-3x speed improvement (with a magnitude less conflicts).

However, I will be monitoring it grow over the next year as we expect an addition ~40,000 samples and ~1,000,000 analyses, and will keep you updated. :rabbit: :+1:

2 Likes

We’ve started to have some performance issues again now that we are in the busy season. While I think updating to a newer version solved some of the issues earlier in the year, I believe I have started to track down a bit more of the cause.

According to Zope’s documentation, it seems like transactions that touch different Persistent objects should not cause conflicts. Therefore, I would expect that WRITE operations to an object (sample, batch, analysis, etc) in one Client would be independent from WRITE operations to an object in another Client.

However, it appears that any 2 write transactions done simultaneously will cause a transaction conflict, and retry the transaction (e.g. User 1 adds Samples to Client A while User 2 saves Analyses in Client B). This causes the site to appear quick when only 1 or 2 users are actively writing data, but the performance exponentially degrades as concurrent operations increase.

We current run a single ZEO Server with 12 clients (10 for users, 2 for debugging/jobs) that has an SSD and about 32GB RAM, so I don’t believe it’s an issue with serving the requests. I’ve also increased the Cache on all of the clients to try to reduce cache times, but these do not truly address the underlying conflict issue.

I’ve looked into changing our ZODB setup to RelStorage, but I am unsure if this would actually avoid these kind of concurrent conflicts. It seems to me like this issue is more related to the relationship between the Archetype content configuration and the Persistence inheritance (and perhaps the custom Schema Extender/Modifiers on this site) :thinking:.

Is anyone familiar with this issue within a Senaite site? Any help or insight would be appreciated!

Hi @faytrow,

Yes, unfortunately these conflict errors on simultaneous writes happen quite often and additionally slow down the write operations in the ZODB.

We figured out, that most of these conflicts happen while the object is being indexed in a ZCText index.

You can inspect this by catching a conflict error traceback and extract the OID of the object that caused the conflict from there.

You can execute the code in the debug console with bin/instance debug:

import rlcompleter, readline
from zope.component.hooks import setSite
from ZODB.utils import p64

readline.parse_and_bind('tab:complete')

portal = app.senaite
setSite(portal)

portal._p_jar[p64(<the-captured-OID>)]

You will likely find a BTree object that contains all the text tokens from a ZCText index.

We already tried to address that for samples in Improve performance for sample listing index by ramonski · Pull Request #2273 · senaite/senaite.core · GitHub and by avoiding to write always back references to the referenced objects: Make `UIDReferenceField` to not keep back-references by default by xispa · Pull Request #2219 · senaite/senaite.core · GitHub

We haven’t tried RelStorage yet, but it might resolve some of the locking issues on concurrent writes. Please share any findings regarding setup and benchmarking if possible.

Some further readings: 22. Scalability and ZEO — Zope 5.8.3 documentation

3 Likes