samba.git/lib, branch ldb-1.2.3

ldb: Release 1.2.3

2017-12-22T09:05:07+00:00

 - Intersect the index from SCOPE_ONELEVEL with the index for the search expression

    This helps ensure we do not have to scan all objects at this level
    which could be very many (one per DNS zone entry).

    However, due to the O(n*m) behaviour in the LDB index code
    we only do this for small numbers of matches on the
    filter tree.

    This behaviour will only be for ldb 1.2 and will not be kept
    long-term in LDB, versions 1.3.1 and above will instead only
    intersect when the more efficient GUID index is in use.

    Finally, disallowDNFilter now applies to SCOPE_ONELEVEL banning
    dn= as a filter string when so configured.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13191

Signed-off-by: Andrew Bartlett 
Reviewed-by: Garming Sam

ldb: Intersect the index from SCOPE_ONELEVEL with the index for the search expression

2017-12-22T09:05:07+00:00

This helps ensure we do not have to scan all objects at this level
which could be very many (one per DNS zone entry).

However, due to the O(n*m) behaviour in list_intersect() for ldb
1.2 and earlier, we only do this for small numbers of matches on the
filter tree.

This behaviour will only be for ldb 1.2 and will not be kept
long-term in LDB, versions 1.3.1 and above will instead only
intersect when the GUID index is in use.

NOTE WELL: the behaviour of disallowDNFilter is enforced
in the index code, so this fixes SCOPE_ONELEVEL to also
honour disallowDNFilter if there @IDXONE is enabled.  Again,
this will change again in 1.3.1 and above.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13191

Signed-off-by: Andrew Bartlett 
Reviewed-by: Garming Sam 
(adapted from ef240aaca0ef693a96726ac2366c454294b87b96 in master)

ldb_tdb: Provide struct ltdb_private to index routines

2017-12-22T09:05:07+00:00

This will make it easier to switch the GUID index mode on and off

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13191

Signed-off-by: Andrew Bartlett 
Reviewed-by: Garming Sam 
(cherry picked from commit 3d952157d72b3a4635f3942449c1727c438c97c6)

pthreadpool: Add a test for the race condition fixed in the last commit

2017-12-14T11:22:18+00:00

Bug: https://bugzilla.samba.org/show_bug.cgi?id=13179
Signed-off-by: Volker Lendecke 
Reviewed-by: Jeremy Allison 
(cherry picked from commit 53f7bbca0451e4f57cdbe8ab4f67f601fe8d40c1)

pthreadpool: Fix starvation after fork

2017-12-14T11:22:18+00:00

After the race is before the race:

1) Create an idle thread
2) Add a job: This won't create a thread anymore
3) Immediately fork

The idle thread will be woken twice before it's actually woken up: Both
pthreadpool_add_job and pthreadpool_prepare_pool call cond_signal, for
different reasons. We must look at pool->prefork_cond first because otherwise
we will end up in a blocking job deep within a fork call, the helper thread
must take its fingers off the condvar as quickly as possible.  This means that
after the fork there's no idle thread around anymore that would pick up the job
submitted in 2). So we must keep the idle threads around across the fork.

The quick solution to re-create one helper thread in pthreadpool_parent has a
fatal flaw: What do we do if that pthread_create call fails? We're deep in an
application calling fork(), and doing fancy signalling from there is really
something we must avoid.

This has one potential performance issue: If we have hundreds of idle threads
(do we ever have that) during the fork, the call to pthread_mutex_lock on the
fork_mutex from pthreadpool_server (the helper thread) will probably cause a
thundering herd when the _parent call unlocks the fork_mutex. The solution for
this to just keep one idle thread around. But this adds code that is not
strictly required functionally for now.

More detailed explanation from Jeremy:

First, understanding the problem the test reproduces:

add a job (num_jobs = 1) -> creates thread to run it.
job finishes, thread sticks around (num_idle = 1).
num_jobs is now zero (initial job finished).

a) Idle thread is now waiting on pool->condvar inside
pthreadpool_server() in pthread_cond_timedwait().

Now, add another job ->

	pthreadpool_add_job()
		-> pthreadpool_put_job()
			This adds the job to the queue.
		Oh, there is an idle thread so don't
		create one, do:

		pthread_cond_signal(&pool->condvar);

		and return.

Now call fork *before* idle thread in (a) wakes from
the signaling of pool->condvar.

In the parent (child is irrelevent):

Go into: pthreadpool_prepare() ->
		pthreadpool_prepare_pool()

		Set the variable to tell idle threads to exit:

		pool->prefork_cond = &prefork_cond;

		then wake them up with:

		pthread_cond_signal(&pool->condvar);

		This does nothing as the idle thread
		is already awoken.

b) Idle thread wakes up and does:

		Reduce idle thread count (num_idle = 0)

		pool->num_idle -= 1;

		Check if we're in the middle of a fork.

		if (pool->prefork_cond != NULL) {

			Yes we are, tell pthreadpool_prepare()
			we are exiting.

			pthread_cond_signal(pool->prefork_cond);

			And exit.

			pthreadpool_server_exit(pool);
			return NULL;
		}

So we come back from the fork in the parent with num_jobs = 1,
a job on the queue but no idle threads - and the code that
creates a new thread on job submission was skipped because
an idle thread existed at point (a).

OK, assuming that the previous explaination is correct, the
fix is to create a new pthreadpool context mutex:

pool->fork_mutex

and in pthreadpool_server(), when an idle thread wakes up and
notices we're in the prepare fork state, it puts itself to
sleep by waiting on the new pool->fork_mutex.

And in pthreadpool_prepare_pool(), instead of waiting for
the idle threads to exit, hold the pool->fork_mutex and
signal each idle thread in turn, and wait for the pool->num_idle
to go to zero - which means they're all blocked waiting on
pool->fork_mutex.

When the parent continues, pthreadpool_parent()
unlocks the pool->fork_mutex and all the previously
'idle' threads wake up (and you mention the thundering
herd problem, which is as you say vanishingly small :-)
and pick up any remaining job.

Bug: https://bugzilla.samba.org/show_bug.cgi?id=13179
Signed-off-by: Volker Lendecke 
Reviewed-by: Jeremy Allison 
(cherry picked from commit f6858505aec9f1004aeaffa83f21e58868749d65)

pthreadpool: Fix deadlock

2017-12-13T09:06:23+00:00

Christof's idea from

https://lists.samba.org/archive/samba-technical/2017-December/124384.html

was that the thread already exited. It could also be that the thread is
not yet idle when the new pthreadpool_add_jobs comes around the corner.

Signed-off-by: Volker Lendecke 
Reviewed-by: Christof Schmitt 

Autobuild-User(master): Christof Schmitt 
Autobuild-Date(master): Wed Dec 13 04:46:12 CET 2017 on sn-devel-144

(cherry picked from commit dfc4670640341761b346065922a62a3e755e9e58)

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13170

pthreadpool: Add test for pthread_create failure

2017-12-13T09:06:23+00:00

This is implemented using cmocka and the __wrap override for
pthread_create.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13170

Signed-off-by: Christof Schmitt 

Autobuild-User(master): Christof Schmitt 
Autobuild-Date(master): Fri Dec  8 13:54:20 CET 2017 on sn-devel-144

(cherry picked from commit 8cdb3995caf7a21d0c27a56a0bf0c1efd5b491e4)

pthreadpool: Undo put_job when returning error

2017-12-13T09:06:23+00:00

When an error is returned to the caller of pthreadpool_add_job, the job
should not be kept in the internal job array. Otherwise the caller might
free the data structure and a later worker thread would still reference
it.

When it is not possible to create a single worker thread, the system
might be out of resources or hitting a configured limit. In this case
fall back to calling the job function synchronously instead of raising
the error to the caller and possibly back to the SMB client.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13170

Signed-off-by: Christof Schmitt 
Reviewed-by: Volker Lendecke 
(cherry picked from commit 065fb5d94d25d19fc85832bb85aa9e379e8551cc)

pthreadpool: Move creating of thread to new function

2017-12-13T09:06:23+00:00

No functional change, but this simplifies error handling.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13170

Signed-off-by: Christof Schmitt 
Reviewed-by: Volker Lendecke 
(cherry picked from commit 949ccc3ea9073a3d38bff28345f644d39177256f)

tevent: version 0.9.34

2017-11-14T11:39:22+00:00

* Remove unused select backend
* Fix a race condition in tevent_threaded_schedule_immediate()
  (bug #13130)

Signed-off-by: Stefan Metzmacher 

Autobuild-User(master): Stefan Metzmacher 
Autobuild-Date(master): Mon Nov 13 18:02:46 CET 2017 on sn-devel-144

(cherry picked from commit 2e573eead96b2e98dd8a15c9c8e470679e530392)