Home

During my consulting practice on various customer accounts i’ve faced some challenges regarding the tuning of the repository caches for increased performance, this post is a result of my personal experiences and it attempts to explain the way Alfresco makes usage of the repository caches.

Note : The information on this post applies only to Alfresco versions 4.1.X.

DISCLAIMER : You should not change the cache values unless you have performance issues and you are absolutely sure about what you are doing. Before changing the cache values you should first gather all the relevant information that will support and justify your changes. The repository caches can have a big positive difference on the performance of your Alfresco repository but they consume Java heap memory. Tuning the caches in a wrong way can make your system irresponsive and may lead to Out of Memory issues. The optimal settings to use on the caches depend on your particular use case and the amount of memory available to your Alfresco server.

The Repository Caches

The alfresco repository features various in-memory caches , they are transaction safe and clusterable. There are 2 levels of caching involved.

Level 1 Cache (cache-context.xml) – The in-transaction caches

Sizes in cache-context.xml are for the in-transaction caches (Level 1 cache) i.e. before it gets flushed to permanent storage. TransactionalCache has to hold any values (read, updated, new, removed) local to the transaction.  On commit, the values are flushed to the level 2 cache (EHCache in 4.1), which makes the data available to other transactions.  Of course, a rollback means the values get thrown away and the shared caches are not modified. This gives Alfresco the power of repeatable read as far as cache values are concerned.

So, if there are a lot of transactions that pull in a lot of cache entries, the transaction-local cache can get full, which is  bad for performance as the only way to guarantee correctness of the shared cache is to clear it. When it comes to site membership evaluation, a large number of ACLs are accessed within single transactions, which is why the transactional cache sizes are larger relative to the shared cache sizes.

Level 2 Cache (ehcache-custom.xml)

The Level 2 (L2) cache provides out-of-transaction caching of Java objects inside the Alfresco system. Alfresco provides support for EHCache. Using EHCache does not restrict the Alfresco system to any particular application server, so it is completely portable.
The L2 cache objects are stored in memory attached to the application scope of the server. Sticky sessions must be used to keep a user that has already established a session on one server for the entire session. By default, the cache replication makes use of RMI to replicate changes to all nodes in the cluster using the Peer Cache Replicator. Each replicated cache member notifies all other cache instances when its content has changed.
Level 2 cache is a technology to speed up database access. When the application makes a database query, it does not have to do a (costly) SQL request if the object is already present in the Level 2 cache. For debugging purposes, you can disable the L2 cache. The database will keep working, but at a slower rate.

If you have issues with the replication of information in clustered systems, that is, the cache cluster test fails, you may want to confirm this by setting the following properties to true in the alfresco-global.properties file as follows :

system.cache.disableMutableSharedCaches=true
system.cache.disableImmutableSharedCaches=true

Default Values for the Caches

Currently, out of the box, a vanilla Alfresco comes setup for approximately  512MB of cache heap memory , that’s the recommended default for a Java heap size of 1GB.  In our days were we have much bigger heaps ( I’ve seen heaps from 4GB up to 54 GB ) and also much bigger numbers in terms of users, concurrency and repository sizes.  This means that the cache defaults on ehcache.xml are designed for dev environments (1G heap) and i personally think they can be tuned on every production environment.

All default cache settings are available in the <configRoot>\alfresco\ehcache-default.xml file, but you should not directly modify this file.

Individual Cache Settings for L2 cache

Each cache is configured in an XML block similar to this:
<cache
name=”org.alfresco.cache.node.rootNodesCache”
maxElementsInMemory=”500″
eternal=”true”
overflowToDisk=”false”
statistics=”false”
/>

name
The name attribute is the name of the cache and generally indicates the type of objects being cached.
maxElementsInMemory
The maxElementsInMemory controls the maximum size of the cache. This value can be changed to tune the size of the cache for your system. Ehcache caches are implemented using a linked-map system, which means that memory is only required for objects that are actually in memory. If you set the maxElementsInMemory to a high value, it will not automatically allocate that number of slots. Instead, they are added to the linked list as required. When maxElementsInMemory is reached, the cache discards the oldest objects before adding new objects.
The cached objects will be garbage collected by means of weak referencing should memory become a problem. It should be noted that some object references are effectively shared by the caches e.g. so the amount of memory used is generally not as high as the approximate value may suggest – but it’s best to error on the side of caution.
timeToIdleSeconds 
timeToIdleSeconds and timeToLiveSeconds control the automatic timeout of cached objects.
overflowToDisk
overflowToDisk controls whether the cache should overflow to disk rather than discarding old objects.
statistics
When set to true and the rest of the tracing mechanism is enabled, the alfresco logs will contain usage statistics of this specific cache.

How to Wisely tune the repository caches for your use-case

There are 2 main files that you can edit/enable to tune your repository caches :  ehcache-custom.xml and cache-context.xml. You’ll find the ehcache-custom.xml.sample file on your shared/classes/alfresco/extension directory under your application server installation folder. To change the default values you need to rename this file to ehcache-custom.xml, perform your changes and restart your application server. When you decide that you need to tune a specific cache, you should do it on both the  ehcache-custom.xml and cache-context.xml files.

We strongly advice you not to tune the caches without tracing their current usage first.

The best way to predict and determine the optimal values for your caches is by using a tracing mechanism that will help you to determine which caches fill up quickly for your particular server use-case.  Take a look below on how to enable the cache tracing mechanism.

TRACING YOUR CACHE SIZES

1 – log4j.properties

Edit your log4j.properties and set the following logging category  to DEBUG to output detailed Ehcache usage information.

org.alfresco.repo.cache.EhCacheTracerJob=DEBUG

To target specific caches, you can even append the cache name or package:

org.alfresco.repo.cache.EhCacheTracerJob.org.alfresco

2 – Override the ehCacheTracerJob bean

The configuration file <configRoot>/alfresco/scheduled-jobs-context.xml contains the
ehCacheTracerJob bean configuration. You will need to override this bean to change the trigger schedule. You will do this by enabling the scheduler property to activate the trigger. To override this bean create a <yourname>-context.xml file on the extensions root folder and provide the overriding for the bean as per the instructions below.

<!– enable DEBUG for ‘org.alfresco.repo.cache.EhCacheTracerJob’ and enable scheduler property to activate —>
<bean id=”ehCacheTracerJob”>
<property name=”jobDetail”>
<bean id=”ehCacheTracerJobDetail”>
<property name=”jobClass”>
<value>org.alfresco.repo.cache.EhCacheTracerJob</value>
</property>
</bean>
</property>
  <!– enable this to activate bean
        <property name=”scheduler”>
            <ref bean=”schedulerFactory” />
        </property>
        –>
<!– start after an hour and repeat hourly –>
<property name=”startDelayMinutes”>
<value>60</value>
</property>
<property name=”repeatIntervalMinutes”>
<value>60</value>
</property>
</bean>

When triggered, the job will collect detailed cache usage statistics and output them to the log/console, depending on how logging has been configured for the server.

3 – Set caches to use statistics

In your ehcache-custom.xml, choose the caches you want to monitor and set the statistics property=true.

<cache
name=”org.alfresco.cache.node.rootNodesCache”
maxElementsInMemory=”250000″
eternal=”true”
overflowToDisk=”false”
statistics=“true”
/>

After making those 3 changes and restarting your server, you should start seeing a detailed output with relevant information in regards to your usage of the repository caches. You should get log traces similar to the ones on the example below.

The following example is from a test of the Alfresco Repository running a simple 150 concurrent user test scenario. Randomly selected from a pool of 1000 test users, a user logs in, views their home space, uploads a small file to the repository and logs out. This test ensures that new objects are continually added to the caches as the new files are added by random users.

Some objects are shared between the caches, so the reported sizes are an overestimate in some cases. Nevertheless, they serve as a useful indication of relative sizes. Do not the last statement that gets logged that clearly indicated the estimated size of your heap that is being consumed by the cache.

09:09:34,458 DEBUG [org.alfresco.repo.cache.EhCacheTracerJob]    Analyzing EHCache:
===>  org.alfresco.repo.domain.hibernate.NodeImpl.sourceNodeAssocs
Hit Count:                        56245 hits            |         Miss Count:         20102 misses
Deep Size:                       19.62 MB              |         Current Count:       5000 entries
Percentage used:            100.00 percent     |         Max Count:           5000 entries
Estimated maximum size:      19.62 MB

09:10:06,099 DEBUG [org.alfresco.repo.cache.EhCacheTracerJob]    Analyzing EHCache:
===>  org.alfresco.repo.domain.hibernate.NodeImpl.targetNodeAssocs
Hit Count:                            56253 hits            |         Miss Count:         20114 misses
Deep Size:                           19.62 MB              |         Current Count:       5000 entries
Percentage used:                100.00 percent     |         Max Count:           5000 entries
Estimated maximum size:      19.62 MB

09:10:06,099 DEBUG [org.alfresco.repo.cache.EhCacheTracerJob]    Analyzing EHCache:
===>  org.alfresco.repo.domain.hibernate.VersionCountImpl
Hit Count:                          0 hits                   |         Miss Count:             0 misses
Deep Size:                         0.00 MB               |         Current Count:          0 entries
Percentage used:              0.00 percent        |         Max Count:            100 entries
Estimated maximum size:        NaN MB

09:10:06,115 DEBUG [org.alfresco.repo.cache.EhCacheTracerJob]    Analyzing EHCache:
===>  org.alfresco.repo.domain.hibernate.NodeAssocImpl
Hit Count:                          0 hits                |         Miss Count:             0 misses
Deep Size:                         0.00 MB            |         Current Count:          0 entries
Percentage used:              0.00 percent     |         Max Count:           1000 entries
Estimated maximum size:        NaN MB


09:10:31,428 DEBUG [org.alfresco.repo.cache.EhCacheTracerJob]    Analyzing EHCache:
===>  permissionsAccessCache
Hit Count:                         2610635 hits        |         Miss Count:       6148423 misses
Deep Size:                        6.02 MB                |         Current Count:      12165 entries
Percentage used:             24.33 percent       |         Max Count:          50000 entries
Estimated maximum size:      24.75 MB

09:10:31,615 DEBUG [org.alfresco.repo.cache.EhCacheTracerJob]    Analyzing EHCache:
===>  PermissionCache
Hit Count:                         9035796 hits        |         Miss Count:      19266775 misses
Deep Size:                        2.55 MB                |         Current Count:       3519 entries
Percentage used:             35.19 percent       |         Max Count:          10000 entries
Estimated maximum size:       7.23 MB

09:10:31,615 DEBUG [org.alfresco.repo.cache.EhCacheTracerJob] EHCaches currently consume 421.84 MB or 28.20% VM size
The criteria to check the cache tracing logs is:

  • (MissCount – CurrentCount) must be as low as possible.
  • (HitCount/MissCount) must be as high as possible.

Estimated maximum size affects the permanent memory taken up by the cache. If the caches grow too large, they may crowd out transient session memory and slow down the system. It is useful to have this running, on occasion, to identify the caches with a low HitCount/MissCount ratio.

An important indicator that you need to increase your caches is when you see messages like the ones below on your alfresco.log file indicating that some specific caches are full.

13:25:12,901 WARN [cache.node.nodesTransactionalCache] Transactional update cache ‘org.alfresco.cache.node.nodesTransactionalCache’ is full (125000).
13:25:14,182 WARN [cache.node.aspectsTransactionalCache] Transactional update cache ‘org.alfresco.cache.node.aspectsTransactionalCache’ is full (65000).
13:25:14,214 WARN [cache.node.propertiesTransactionalCache] Transactional update cache ‘org.alfresco.cache.node.propertiesTransactionalCache’ is full (65000).

After analysing your tracing results and your alfresco log file , if you decide to perform a repository cache tuning, your should enable for editing both the ehcache-custom.xmland cache-context.xml.

I strongly advice you not to change the original cache-context.xml file directly, but to create a new context file where you override the beans that you need. You find the original cache-context files on the alfresco.war webapp, under /classes/alfresco/cache-context.xml. The tuning of a specific cache should occur on both the cache-context file and the ehcache-custom.xml file. For your reference, i’m attaching to this post 2 sample cache configuration files with settings that were  required for to handle a large group structure environment.

custom-cache-context.xml ehcache-custom.xml

Individual Caches Information

This section shows details on the most important repository caches with a brief definition on their purpose.

Default Cache
defaultCache This cache is for when someone forgets to write a cache sizing snippet.
Store Caches
rootNodesCache Primary root nodes by store.  Increase for Multi Tenancy
allRootNodesCache All root nodes by store.  Increase for Multi Tenancy
Nodes Caches
nodesCache Bi-direction mapping between NodeRef and ID.
aspectsCache Node aspects by node ID.  Size relative to nodesCache
propertiesCache Node properties by node ID.  Size relative to nodesCache
childByNameCache Child node IDs cached by parent node ID and cm:name.  Used for direct name lookups and especially useful for heavy CIFS use.
contentDataCache cm:content ContentData storage.  Size according to content in system.
Permissions and ACLs: (Mostly used in Alfresco Share)
authorityToChildAuthorityCache Size according to total number of users accessing system
authorityToChildAuthorityCache Members of each group. Size according to number of groups, incl. site groups
zoneToAuthorityCache Authorities in each zone. Size according to zones.
authenticationCache Scale it according to the number of users, not concurrency, because Share sites will force all of them to be hit regardless of who logs in.
authorityCache NodeRef of the authority.  Size according to total authorities
permissionsAccessCache Size according to total authorities
readersCache Who can read a node, Size according to total authorities.
readersDeniedCache Who can’t read a node,  Size according to total authorities.
nodeOwnerCache cm:ownable mapping, needed during permission checks
personCache Size according to the number of people accessing the system
aclCache ACL mapped by Access control list properties
aclEntityCache DAO level cache bi-directional ID-ACL mapping.
Other Caches 
propertyValueCache Caches values stored for auditing.  Size according to frequency and size of audit queries and audit value generation
immutableEntityCache QNames, etc.  Size according to static model sizes
tagscopeSummaryCache Size according to tag use.  Stores rollups of tags

AVM Caches are related to a deprecated part of Alfresco so you shouldn’t need to tune the following caches :

  • avmEntityCache
  • avmVersionRootEntityCache
  • avmNodeCache
  • avmStoreCache
  • avmNodeAspectsCache

Conclusion

Repository caches play an important role on the repository performance and they should be tuned wisely. Always use the tracing method before you decide to tune your caches and don’t forget to check your logs for signs that can help you on decide that you need to tune your caches.

I hope you enjoyed this article, stay tuned for more information from the field.

 

Publicités

Laisser un commentaire

Choisissez une méthode de connexion pour poster votre commentaire:

Logo WordPress.com

Vous commentez à l'aide de votre compte WordPress.com. Déconnexion / Changer )

Image Twitter

Vous commentez à l'aide de votre compte Twitter. Déconnexion / Changer )

Photo Facebook

Vous commentez à l'aide de votre compte Facebook. Déconnexion / Changer )

Photo Google+

Vous commentez à l'aide de votre compte Google+. Déconnexion / Changer )

Connexion à %s