During my consulting practice on various customer accounts i’ve faced some challenges regarding the tuning of the repository caches for increased performance, this post is a result of my personal experiences and it attempts to explain the way Alfresco makes usage of the repository caches.
Note : The information on this post applies only to Alfresco versions 4.1.X.
DISCLAIMER : You should not change the cache values unless you have performance issues and you are absolutely sure about what you are doing. Before changing the cache values you should first gather all the relevant information that will support and justify your changes. The repository caches can have a big positive difference on the performance of your Alfresco repository but they consume Java heap memory. Tuning the caches in a wrong way can make your system irresponsive and may lead to Out of Memory issues. The optimal settings to use on the caches depend on your particular use case and the amount of memory available to your Alfresco server.
The Repository Caches
The alfresco repository features various in-memory caches , they are transaction safe and clusterable. There are 2 levels of caching involved.
Level 1 Cache (cache-context.xml) – The in-transaction caches
Sizes in cache-context.xml are for the in-transaction caches (Level 1 cache) i.e. before it gets flushed to permanent storage. TransactionalCache has to hold any values (read, updated, new, removed) local to the transaction. On commit, the values are flushed to the level 2 cache (EHCache in 4.1), which makes the data available to other transactions. Of course, a rollback means the values get thrown away and the shared caches are not modified. This gives Alfresco the power of repeatable read as far as cache values are concerned.
So, if there are a lot of transactions that pull in a lot of cache entries, the transaction-local cache can get full, which is bad for performance as the only way to guarantee correctness of the shared cache is to clear it. When it comes to site membership evaluation, a large number of ACLs are accessed within single transactions, which is why the transactional cache sizes are larger relative to the shared cache sizes.
Level 2 Cache (ehcache-custom.xml)
The Level 2 (L2) cache provides out-of-transaction caching of Java objects inside the Alfresco system. Alfresco provides support for EHCache. Using EHCache does not restrict the Alfresco system to any particular application server, so it is completely portable.
The L2 cache objects are stored in memory attached to the application scope of the server. Sticky sessions must be used to keep a user that has already established a session on one server for the entire session. By default, the cache replication makes use of RMI to replicate changes to all nodes in the cluster using the Peer Cache Replicator. Each replicated cache member notifies all other cache instances when its content has changed.
Level 2 cache is a technology to speed up database access. When the application makes a database query, it does not have to do a (costly) SQL request if the object is already present in the Level 2 cache. For debugging purposes, you can disable the L2 cache. The database will keep working, but at a slower rate.
If you have issues with the replication of information in clustered systems, that is, the cache cluster test fails, you may want to confirm this by setting the following properties to true in the alfresco-global.properties file as follows :
system.cache.disableMutableSharedCaches=true
system.cache.disableImmutableSharedCaches=true
Default Values for the Caches
Currently, out of the box, a vanilla Alfresco comes setup for approximately 512MB of cache heap memory , that’s the recommended default for a Java heap size of 1GB. In our days were we have much bigger heaps ( I’ve seen heaps from 4GB up to 54 GB ) and also much bigger numbers in terms of users, concurrency and repository sizes. This means that the cache defaults on ehcache.xml are designed for dev environments (1G heap) and i personally think they can be tuned on every production environment.
All default cache settings are available in the <configRoot>\alfresco\ehcache-default.xml file, but you should not directly modify this file.
Individual Cache Settings for L2 cache
Each cache is configured in an XML block similar to this:
<cache
name=”org.alfresco.cache.node.rootNodesCache”
maxElementsInMemory=”500″
eternal=”true”
overflowToDisk=”false”
statistics=”false”
/>
- name
- The name attribute is the name of the cache and generally indicates the type of objects being cached.
- maxElementsInMemory
- The maxElementsInMemory controls the maximum size of the cache. This value can be changed to tune the size of the cache for your system. Ehcache caches are implemented using a linked-map system, which means that memory is only required for objects that are actually in memory. If you set the maxElementsInMemory to a high value, it will not automatically allocate that number of slots. Instead, they are added to the linked list as required. When maxElementsInMemory is reached, the cache discards the oldest objects before adding new objects.
- The cached objects will be garbage collected by means of weak referencing should memory become a problem. It should be noted that some object references are effectively shared by the caches e.g. so the amount of memory used is generally not as high as the approximate value may suggest – but it’s best to error on the side of caution.
- timeToIdleSeconds
- timeToIdleSeconds and timeToLiveSeconds control the automatic timeout of cached objects.
- overflowToDisk
- overflowToDisk controls whether the cache should overflow to disk rather than discarding old objects.
- statistics
- When set to true and the rest of the tracing mechanism is enabled, the alfresco logs will contain usage statistics of this specific cache.
How to Wisely tune the repository caches for your use-case
There are 2 main files that you can edit/enable to tune your repository caches : ehcache-custom.xml and cache-context.xml. You’ll find the ehcache-custom.xml.sample file on your shared/classes/alfresco/extension directory under your application server installation folder. To change the default values you need to rename this file to ehcache-custom.xml, perform your changes and restart your application server. When you decide that you need to tune a specific cache, you should do it on both the ehcache-custom.xml and cache-context.xml files.
We strongly advice you not to tune the caches without tracing their current usage first.
The best way to predict and determine the optimal values for your caches is by using a tracing mechanism that will help you to determine which caches fill up quickly for your particular server use-case. Take a look below on how to enable the cache tracing mechanism.
TRACING YOUR CACHE SIZES
1 – log4j.properties
Edit your log4j.properties and set the following logging category to DEBUG to output detailed Ehcache usage information.
org.alfresco.repo.cache.EhCacheTracerJob=DEBUG
To target specific caches, you can even append the cache name or package:
org.alfresco.repo.cache.EhCacheTracerJob.org.alfresco
2 – Override the ehCacheTracerJob bean
The configuration file <configRoot>/alfresco/scheduled-jobs-context.xml contains the
ehCacheTracerJob bean configuration. You will need to override this bean to change the trigger schedule. You will do this by enabling the scheduler property to activate the trigger. To override this bean create a <yourname>-context.xml file on the extensions root folder and provide the overriding for the bean as per the instructions below.
<!– enable DEBUG for ‘org.alfresco.repo.cache.EhCacheTracerJob’ and enable scheduler property to activate —>
<bean id=”ehCacheTracerJob”>
<property name=”jobDetail”>
<bean id=”ehCacheTracerJobDetail”>
<property name=”jobClass”>
<value>org.alfresco.repo.cache.EhCacheTracerJob</value>
</property>
</bean>
</property>
<!– enable this to activate bean
<property name=”scheduler”>
<ref bean=”schedulerFactory” />
</property>
–>
<!– start after an hour and repeat hourly –>
<property name=”startDelayMinutes”>
<value>60</value>
</property>
<property name=”repeatIntervalMinutes”>
<value>60</value>
</property>
</bean>
When triggered, the job will collect detailed cache usage statistics and output them to the log/console, depending on how logging has been configured for the server.
3 – Set caches to use statistics
In your ehcache-custom.xml, choose the caches you want to monitor and set the statistics property=true.
<cache
name=”org.alfresco.cache.node.rootNodesCache”
maxElementsInMemory=”250000″
eternal=”true”
overflowToDisk=”false”
statistics=“true”
/>
After making those 3 changes and restarting your server, you should start seeing a detailed output with relevant information in regards to your usage of the repository caches. You should get log traces similar to the ones on the example below.
The following example is from a test of the Alfresco Repository running a simple 150 concurrent user test scenario. Randomly selected from a pool of 1000 test users, a user logs in, views their home space, uploads a small file to the repository and logs out. This test ensures that new objects are continually added to the caches as the new files are added by random users.
Some objects are shared between the caches, so the reported sizes are an overestimate in some cases. Nevertheless, they serve as a useful indication of relative sizes. Do not the last statement that gets logged that clearly indicated the estimated size of your heap that is being consumed by the cache.
09:09:34,458 DEBUG [org.alfresco.repo.cache.EhCacheTracerJob] Analyzing EHCache:
===> org.alfresco.repo.domain.hibernate.NodeImpl.sourceNodeAssocs
Hit Count: 56245 hits | Miss Count: 20102 misses
Deep Size: 19.62 MB | Current Count: 5000 entries
Percentage used: 100.00 percent | Max Count: 5000 entries
Estimated maximum size: 19.62 MB
09:10:06,099 DEBUG [org.alfresco.repo.cache.EhCacheTracerJob] Analyzing EHCache:
===> org.alfresco.repo.domain.hibernate.NodeImpl.targetNodeAssocs
Hit Count: 56253 hits | Miss Count: 20114 misses
Deep Size: 19.62 MB | Current Count: 5000 entries
Percentage used: 100.00 percent | Max Count: 5000 entries
Estimated maximum size: 19.62 MB
09:10:06,099 DEBUG [org.alfresco.repo.cache.EhCacheTracerJob] Analyzing EHCache:
===> org.alfresco.repo.domain.hibernate.VersionCountImpl
Hit Count: 0 hits | Miss Count: 0 misses
Deep Size: 0.00 MB | Current Count: 0 entries
Percentage used: 0.00 percent | Max Count: 100 entries
Estimated maximum size: NaN MB
09:10:06,115 DEBUG [org.alfresco.repo.cache.EhCacheTracerJob] Analyzing EHCache:
===> org.alfresco.repo.domain.hibernate.NodeAssocImpl
Hit Count: 0 hits | Miss Count: 0 misses
Deep Size: 0.00 MB | Current Count: 0 entries
Percentage used: 0.00 percent | Max Count: 1000 entries
Estimated maximum size: NaN MB
…
09:10:31,428 DEBUG [org.alfresco.repo.cache.EhCacheTracerJob] Analyzing EHCache:
===> permissionsAccessCache
Hit Count: 2610635 hits | Miss Count: 6148423 misses
Deep Size: 6.02 MB | Current Count: 12165 entries
Percentage used: 24.33 percent | Max Count: 50000 entries
Estimated maximum size: 24.75 MB
09:10:31,615 DEBUG [org.alfresco.repo.cache.EhCacheTracerJob] Analyzing EHCache:
===> PermissionCache
Hit Count: 9035796 hits | Miss Count: 19266775 misses
Deep Size: 2.55 MB | Current Count: 3519 entries
Percentage used: 35.19 percent | Max Count: 10000 entries
Estimated maximum size: 7.23 MB
09:10:31,615 DEBUG [org.alfresco.repo.cache.EhCacheTracerJob] EHCaches currently consume 421.84 MB or 28.20% VM size
The criteria to check the cache tracing logs is:
- (MissCount – CurrentCount) must be as low as possible.
- (HitCount/MissCount) must be as high as possible.
Estimated maximum size affects the permanent memory taken up by the cache. If the caches grow too large, they may crowd out transient session memory and slow down the system. It is useful to have this running, on occasion, to identify the caches with a low HitCount/MissCount ratio.
An important indicator that you need to increase your caches is when you see messages like the ones below on your alfresco.log file indicating that some specific caches are full.
13:25:12,901 WARN [cache.node.nodesTransactionalCache] Transactional update cache ‘org.alfresco.cache.node.nodesTransactionalCache’ is full (125000).
13:25:14,182 WARN [cache.node.aspectsTransactionalCache] Transactional update cache ‘org.alfresco.cache.node.aspectsTransactionalCache’ is full (65000).
13:25:14,214 WARN [cache.node.propertiesTransactionalCache] Transactional update cache ‘org.alfresco.cache.node.propertiesTransactionalCache’ is full (65000).
After analysing your tracing results and your alfresco log file , if you decide to perform a repository cache tuning, your should enable for editing both the ehcache-custom.xmland cache-context.xml.
I strongly advice you not to change the original cache-context.xml file directly, but to create a new context file where you override the beans that you need. You find the original cache-context files on the alfresco.war webapp, under /classes/alfresco/cache-context.xml. The tuning of a specific cache should occur on both the cache-context file and the ehcache-custom.xml file. For your reference, i’m attaching to this post 2 sample cache configuration files with settings that were required for to handle a large group structure environment.
custom-cache-context.xml ehcache-custom.xml
Individual Caches Information
This section shows details on the most important repository caches with a brief definition on their purpose.
| Default Cache | |
| defaultCache | This cache is for when someone forgets to write a cache sizing snippet. |
| Store Caches | |
| rootNodesCache | Primary root nodes by store. Increase for Multi Tenancy |
| allRootNodesCache | All root nodes by store. Increase for Multi Tenancy |
| Nodes Caches | |
| nodesCache | Bi-direction mapping between NodeRef and ID. |
| aspectsCache | Node aspects by node ID. Size relative to nodesCache |
| propertiesCache | Node properties by node ID. Size relative to nodesCache |
| childByNameCache | Child node IDs cached by parent node ID and cm:name. Used for direct name lookups and especially useful for heavy CIFS use. |
| contentDataCache | cm:content ContentData storage. Size according to content in system. |
| Permissions and ACLs: (Mostly used in Alfresco Share) | |
| authorityToChildAuthorityCache | Size according to total number of users accessing system |
| authorityToChildAuthorityCache | Members of each group. Size according to number of groups, incl. site groups |
| zoneToAuthorityCache | Authorities in each zone. Size according to zones. |
| authenticationCache | Scale it according to the number of users, not concurrency, because Share sites will force all of them to be hit regardless of who logs in. |
| authorityCache | NodeRef of the authority. Size according to total authorities |
| permissionsAccessCache | Size according to total authorities |
| readersCache | Who can read a node, Size according to total authorities. |
| readersDeniedCache | Who can’t read a node, Size according to total authorities. |
| nodeOwnerCache | cm:ownable mapping, needed during permission checks |
| personCache | Size according to the number of people accessing the system |
| aclCache | ACL mapped by Access control list properties |
| aclEntityCache | DAO level cache bi-directional ID-ACL mapping. |
| Other Caches | |
| propertyValueCache | Caches values stored for auditing. Size according to frequency and size of audit queries and audit value generation |
| immutableEntityCache | QNames, etc. Size according to static model sizes |
| tagscopeSummaryCache | Size according to tag use. Stores rollups of tags |
AVM Caches are related to a deprecated part of Alfresco so you shouldn’t need to tune the following caches :
- avmEntityCache
- avmVersionRootEntityCache
- avmNodeCache
- avmStoreCache
- avmNodeAspectsCache
Conclusion
Repository caches play an important role on the repository performance and they should be tuned wisely. Always use the tracing method before you decide to tune your caches and don’t forget to check your logs for signs that can help you on decide that you need to tune your caches.
I hope you enjoyed this article, stay tuned for more information from the field.
vienergie 