Class ConcurrentMergeScheduler
- All Implemented Interfaces:
Closeable,AutoCloseable
MergeScheduler that runs each merge using a separate thread.
Specify the max number of threads that may run at once, and the maximum number of simultaneous
merges with setMaxMergesAndThreads(int, int).
If the number of merges exceeds the max number of threads then the largest merges are paused until one of the smaller merges completes.
If more than getMaxMergeCount() merges are requested then this class will forcefully
throttle the incoming threads by pausing until one more merges complete.
This class sets defaults based on Java's view of the cpu count, and it assumes a solid state
disk (or similar). If you have a spinning disk and want to maximize performance, use setDefaultMaxMergesAndThreads(boolean).
-
Nested Class Summary
Nested ClassesModifier and TypeClassDescriptionprotected classRuns a merge thread to execute a single merge, then exits.Nested classes/interfaces inherited from class org.apache.lucene.index.MergeScheduler
MergeScheduler.MergeSource -
Field Summary
FieldsModifier and TypeFieldDescriptionstatic final intDynamic default formaxThreadCountandmaxMergeCount, based on CPU core count.static final StringUsed for testing.protected org.apache.lucene.index.ConcurrentMergeScheduler.CachedExecutorThe executor provided for intra-merge parallelizationprotected intHow manyConcurrentMergeScheduler.MergeThreads have kicked off (this is use to name them).protected final List<ConcurrentMergeScheduler.MergeThread> List of currently activeConcurrentMergeScheduler.MergeThreads.protected doubleCurrent IO writes throttle rateFields inherited from class org.apache.lucene.index.MergeScheduler
infoStream -
Constructor Summary
ConstructorsConstructorDescriptionSole constructor, with all settings set to default values. -
Method Summary
Modifier and TypeMethodDescriptionvoidclose()Close this MergeScheduler.voidTurn off auto IO throttling.protected voiddoMerge(MergeScheduler.MergeSource mergeSource, MergePolicy.OneMerge merge) Does the actual merge, by callingMergeScheduler.MergeSource.merge(org.apache.lucene.index.MergePolicy.OneMerge)protected voiddoStall()Called frommaybeStall(org.apache.lucene.index.MergeScheduler.MergeSource)to pause the calling thread for a bit.voidTurn on dynamic IO throttling, to adaptively rate limit writes bytes/sec to the minimal rate necessary so merges do not fall behind.booleanReturns true if auto IO throttling is currently enabled.doubleGet the per-merge IO throttle rate for forced merges.Provides an executor for parallelism during a single merge operation.doubleReturns the currently set per-merge IO writes rate limit, ifenableAutoIOThrottle()was called, elseDouble.POSITIVE_INFINITY.intintReturnsmaxThreadCount.protected ConcurrentMergeScheduler.MergeThreadgetMergeThread(MergeScheduler.MergeSource mergeSource, MergePolicy.OneMerge merge) Create and return a new MergeThreadprotected voidCalled when an exception is hit in a background merge threadprotected booleanmaybeStall(MergeScheduler.MergeSource mergeSource) This is invoked bymerge(org.apache.lucene.index.MergeScheduler.MergeSource, org.apache.lucene.index.MergeTrigger)to possibly stall the incoming thread when there are too many merges running or pending.voidmerge(MergeScheduler.MergeSource mergeSource, MergeTrigger trigger) Run the merges provided byMergeScheduler.MergeSource.getNextMerge().intReturns the number of merge threads that are alive, ignoring the calling thread if it is a merge thread.voidsetDefaultMaxMergesAndThreads(boolean spins) Sets max merges and threads to proper defaults for rotational or non-rotational storage.voidsetForceMergeMBPerSec(double v) Set the per-merge IO throttle rate for forced merges (default:Double.POSITIVE_INFINITY).voidsetMaxMergesAndThreads(int maxMergeCount, int maxThreadCount) Expert: directly set the maximum number of merge threads and simultaneous merges allowed.voidsync()Wait for any running merge threads to finish.protected voidSubclass can override to tweak targetMBPerSec.toString()protected voidCalled whenever the running merges have changed, to set merge IO limits.wrapForMerge(MergePolicy.OneMerge merge, Directory in) Wraps the incomingDirectoryso that we can merge-throttle it usingRateLimitedIndexOutput.Methods inherited from class org.apache.lucene.index.MergeScheduler
message, verbose
-
Field Details
-
AUTO_DETECT_MERGES_AND_THREADS
public static final int AUTO_DETECT_MERGES_AND_THREADSDynamic default formaxThreadCountandmaxMergeCount, based on CPU core count.maxThreadCountis set tomax(1, min(4, cpuCoreCount/2)).maxMergeCountis set tomaxThreadCount + 5.- See Also:
-
DEFAULT_CPU_CORE_COUNT_PROPERTY
Used for testing.- See Also:
- NOTE: This API is for internal purposes only and might change in incompatible ways in the next release.
-
mergeThreads
List of currently activeConcurrentMergeScheduler.MergeThreads. -
mergeThreadCounter
protected int mergeThreadCounterHow manyConcurrentMergeScheduler.MergeThreads have kicked off (this is use to name them). -
targetMBPerSec
protected double targetMBPerSecCurrent IO writes throttle rate -
intraMergeExecutor
protected org.apache.lucene.index.ConcurrentMergeScheduler.CachedExecutor intraMergeExecutorThe executor provided for intra-merge parallelization
-
-
Constructor Details
-
ConcurrentMergeScheduler
public ConcurrentMergeScheduler()Sole constructor, with all settings set to default values.
-
-
Method Details
-
setMaxMergesAndThreads
public void setMaxMergesAndThreads(int maxMergeCount, int maxThreadCount) Expert: directly set the maximum number of merge threads and simultaneous merges allowed.- Parameters:
maxMergeCount- the max # simultaneous merges that are allowed. If a merge is necessary yet we already have this many threads running, the incoming thread (that is calling add/updateDocument) will block until a merge thread has completed. Note that we will only run the smallestmaxThreadCountmerges at a time.maxThreadCount- the max # simultaneous merge threads that should be running at once. This must be <=maxMergeCount
-
setDefaultMaxMergesAndThreads
public void setDefaultMaxMergesAndThreads(boolean spins) Sets max merges and threads to proper defaults for rotational or non-rotational storage.- Parameters:
spins- true to set defaults best for traditional rotatational storage (spinning disks), else false (e.g. for solid-state disks)
-
setForceMergeMBPerSec
public void setForceMergeMBPerSec(double v) Set the per-merge IO throttle rate for forced merges (default:Double.POSITIVE_INFINITY). -
getForceMergeMBPerSec
public double getForceMergeMBPerSec()Get the per-merge IO throttle rate for forced merges. -
enableAutoIOThrottle
public void enableAutoIOThrottle()Turn on dynamic IO throttling, to adaptively rate limit writes bytes/sec to the minimal rate necessary so merges do not fall behind. By default this is disabled and writes are not rate-limited. -
disableAutoIOThrottle
public void disableAutoIOThrottle()Turn off auto IO throttling.- See Also:
-
getAutoIOThrottle
public boolean getAutoIOThrottle()Returns true if auto IO throttling is currently enabled. -
getIORateLimitMBPerSec
public double getIORateLimitMBPerSec()Returns the currently set per-merge IO writes rate limit, ifenableAutoIOThrottle()was called, elseDouble.POSITIVE_INFINITY. -
getMaxThreadCount
public int getMaxThreadCount()ReturnsmaxThreadCount.- See Also:
-
getMaxMergeCount
public int getMaxMergeCount() -
getIntraMergeExecutor
Description copied from class:MergeSchedulerProvides an executor for parallelism during a single merge operation. By default, the method returns aSameThreadExecutorServicewhere all intra-merge actions occur in their calling thread.- Overrides:
getIntraMergeExecutorin classMergeScheduler
-
wrapForMerge
Description copied from class:MergeSchedulerWraps the incomingDirectoryso that we can merge-throttle it usingRateLimitedIndexOutput.- Overrides:
wrapForMergein classMergeScheduler
-
updateMergeThreads
protected void updateMergeThreads()Called whenever the running merges have changed, to set merge IO limits. This method sorts the merge threads by their merge size in descending order and then pauses/unpauses threads from first to last -- that way, smaller merges are guaranteed to run before larger ones. -
close
Description copied from class:MergeSchedulerClose this MergeScheduler.- Specified by:
closein interfaceAutoCloseable- Specified by:
closein interfaceCloseable- Overrides:
closein classMergeScheduler- Throws:
IOException
-
sync
public void sync()Wait for any running merge threads to finish. This call is not interruptible as used byclose(). -
mergeThreadCount
public int mergeThreadCount()Returns the number of merge threads that are alive, ignoring the calling thread if it is a merge thread. Note that this number is ≤mergeThreadssize.- NOTE: This API is for internal purposes only and might change in incompatible ways in the next release.
-
merge
Description copied from class:MergeSchedulerRun the merges provided byMergeScheduler.MergeSource.getNextMerge().- Specified by:
mergein classMergeScheduler- Parameters:
mergeSource- theIndexWriterto obtain the merges from.trigger- theMergeTriggerthat caused this merge to happen- Throws:
IOException
-
maybeStall
This is invoked bymerge(org.apache.lucene.index.MergeScheduler.MergeSource, org.apache.lucene.index.MergeTrigger)to possibly stall the incoming thread when there are too many merges running or pending. The default behavior is to force this thread, which is producing too many segments for merging to keep up, to wait until merges catch up. Applications that can take other less drastic measures, such as limiting how many threads are allowed to index, can do nothing here and throttle elsewhere.If this method wants to stall but the calling thread is a merge thread, it should return false to tell caller not to kick off any new merges.
-
doStall
protected void doStall()Called frommaybeStall(org.apache.lucene.index.MergeScheduler.MergeSource)to pause the calling thread for a bit. -
doMerge
protected void doMerge(MergeScheduler.MergeSource mergeSource, MergePolicy.OneMerge merge) throws IOException Does the actual merge, by callingMergeScheduler.MergeSource.merge(org.apache.lucene.index.MergePolicy.OneMerge)- Throws:
IOException
-
getMergeThread
protected ConcurrentMergeScheduler.MergeThread getMergeThread(MergeScheduler.MergeSource mergeSource, MergePolicy.OneMerge merge) throws IOException Create and return a new MergeThread- Throws:
IOException
-
handleMergeException
Called when an exception is hit in a background merge thread -
toString
-
targetMBPerSecChanged
protected void targetMBPerSecChanged()Subclass can override to tweak targetMBPerSec.
-