samba.git/source3/lib/dbwrap, branch talloc-2.3.3

dbwrap_ctdb: Remove "tryonly" from fetch_locked_internal()

2021-06-04T16:47:34+00:00

Signed-off-by: Volker Lendecke 
Reviewed-by: Jeremy Allison

dbwrap: Remove "db_context->try_fetch_locked()" fn pointer

2021-06-04T16:47:34+00:00

Signed-off-by: Volker Lendecke 
Reviewed-by: Jeremy Allison

lib: give global_contexts.c its own header file

2021-01-08T20:31:33+00:00

It's a bit shocking how many references we have to global
contexts. Make this a bit more obvious.

Signed-off-by: Volker Lendecke 
Reviewed-by: Ralph Boehme 
Reviewed-by: Jeremy Allison

s3:dbwrap_watch: avoid recursion into dbwrap_do_locked() from dbwrap_watched_do_locked_{storev,delete}()

2020-07-08T11:02:39+00:00

This avoids a lot of overhead!

Using smbtorture3 //foo/bar -U% local-g-lock-ping-pong -o 500000
under valgrind --tool=callgrind...

This change replaces this:

 6,877,542,529  PROGRAM TOTALS

   590,000,773  lib/tdb/common/lock.c:tdb_lock_list
   479,000,608  lib/tdb/common/lock.c:tdb_unlock
   446,500,532  lib/tdb/common/io.c:tdb_read
   364,000,824  lib/tdb/common/hash.c:tdb_jenkins_hash
   285,000,532  lib/tdb/common/io.c:tdb_write
   262,054,669  /x86_64/multiarch/memmove-vec-unaligned-erms.S:__memcpy_avx_unaligned_erms
   206,500,496  lib/tdb/common/mutex.c:tdb_mutex_lock
   193,000,176  lib/tdb/common/tdb.c:tdb_find
   160,000,256  lib/talloc/talloc.c:_talloc_get_type_abort
   148,500,297  lib/tdb/common/tdb.c:tdb_storev
   140,000,196  lib/tdb/common/lock.c:tdb_lock
   130,000,858  lib/util/debug.c:debuglevel_get_class
   128,003,722  lib/talloc/talloc.c:_talloc_free
   128,000,118  lib/tdb/common/tdb.c:tdb_parse_record
   126,000,576  lib/tdb/common/lock.c:tdb_brlock.part.3
   121,000,272  lib/tdb/common/mutex.c:tdb_mutex_unlock
   118,000,225  /nptl/pthread_mutex_lock.c:__pthread_mutex_lock_full
   112,750,222  lib/tdb/common/freelist.c:tdb_allocate_from_freelist
   108,500,168  lib/tdb/common/io.c:tdb_ofs_read
   102,500,000  lib/tdb/common/io.c:tdb_parse_data

by this:

 5,706,522,398  PROGRAM TOTALS

   434,000,617  lib/tdb/common/lock.c:tdb_lock_list
   389,500,494  lib/tdb/common/io.c:tdb_read
   359,000,488  lib/tdb/common/lock.c:tdb_unlock
   285,000,532  lib/tdb/common/io.c:tdb_write
   237,554,655  /x86_64/multiarch/memmove-vec-unaligned-erms.S:__memcpy_avx_unaligned_erms
   208,000,668  lib/tdb/common/hash.c:tdb_jenkins_hash
   206,500,496  lib/tdb/common/mutex.c:tdb_mutex_lock
   160,000,256  lib/talloc/talloc.c:_talloc_get_type_abort
   148,500,297  lib/tdb/common/tdb.c:tdb_storev
   136,000,132  lib/tdb/common/tdb.c:tdb_find
   130,000,858  lib/util/debug.c:debuglevel_get_class
   126,000,576  lib/tdb/common/lock.c:tdb_brlock.part.3
   121,000,272  lib/tdb/common/mutex.c:tdb_mutex_unlock
   118,000,225  /nptl/pthread_mutex_lock.c:__pthread_mutex_lock_full
   112,750,222  lib/tdb/common/freelist.c:tdb_allocate_from_freelist
   112,000,168  lib/tdb/common/lock.c:tdb_lock
    94,500,154  lib/tdb/common/io.c:tdb_ofs_read
    94,000,188  /nptl/pthread_mutex_unlock.c:__pthread_mutex_unlock_full
    86,000,086  lib/dbwrap/dbwrap.c:dbwrap_lock_order_lock
    83,000,083  lib/dbwrap/dbwrap_tdb.c:db_tdb_do_locked

time smbtorture3 //foo/bar -U% local-g-lock-ping-pong -o 5000000

gives:

  902834 locks/sec

 real    0m11,103s
 user    0m8,233s
 sys     0m2,868s

vs.

 1037262 locks/sec

 real    0m9,685s
 user    0m6,788s
 sys     0m2,896s

Signed-off-by: Stefan Metzmacher 
Reviewed-by: Volker Lendecke 

Autobuild-User(master): Volker Lendecke 
Autobuild-Date(master): Wed Jul  8 11:02:39 UTC 2020 on sn-devel-184

lib: Allow DBWRAP_LOCK_ORDER_NONE in db_open()

2020-05-15T00:48:32+00:00

locking.tdb will not have a LOCK_ORDER anymore, this will be done by
the code in g_lock.c. We need to allow opening a database with dbwrap
without having a lock order.

Signed-off-by: Volker Lendecke 
Reviewed-by: Jeremy Allison

lib: Fix a comment

2020-05-05T11:48:38+00:00

The "deleted bit" went away with 341223a00596e

Signed-off-by: Volker Lendecke 
Reviewed-by: Ralph Boehme

lib: Use ctdbd_req_send/recv in ctdb_parse_send/recv

2020-04-28T09:08:40+00:00

Signed-off-by: Volker Lendecke 
Reviewed-by: Ralph Boehme

dbwrap_watch: Set rec->value_valid while returning nested share_mode_do_locked()

2020-04-21T17:37:43+00:00

As reported on samba-technical by Rouven WEILER :
https://lists.samba.org/archive/samba-technical/2020-April/135116.html

Following backtrace was observed with vfs_fruit for time machine backup:

[2020/04/10 08:00:38.107917,  0] ../../lib/dbwrap/dbwrap.c:82(dbwrap_record_get_value)
  PANIC: assert failed at ../../lib/dbwrap/dbwrap.c(82): rec->value_valid
[2020/04/10 08:00:38.108499,  0] ../../source3/lib/util.c:830(smb_panic_s3)
  PANIC (pid 3427): assert failed: rec->value_valid
[2020/04/10 08:00:38.109541,  0] ../../lib/util/fault.c:265(log_stack_trace)
  BACKTRACE: 37 stack frames:
   #0 /usr/lib/samba/amd64/libsamba-util.so.0.0.1'log_stack_trace+0x26 [0xfffffd7fee51de66]
   #1 /usr/lib/samba/amd64/libsmbconf.so.0'smb_panic_s3+0x26 [0xfffffd7fedf5a596]
   #2 /usr/lib/samba/amd64/libsamba-util.so.0.0.1'smb_panic+0x1f [0xfffffd7fee51df3f]
   #3 /usr/lib/samba/private/amd64/libdbwrap-samba4.so'dbwrap_record_get_value+0x2a [0xfffffd7feccb627a]
   #4 /usr/lib/samba/private/amd64/libsmbd-base-samba4.so'get_share_mode_lock+0x109 [0xfffffd7fee7195c9]
   #5 /usr/lib/samba/private/amd64/libsmbd-base-samba4.so'smbd_contend_level2_oplocks_begin+0xa1 [0xfffffd7fee7f7761]
   #6 /usr/lib/samba/private/amd64/libsmbd-base-samba4.so'brl_lock+0x635 [0xfffffd7fee710f45]
   #7 /usr/lib/samba/private/amd64/libsmbd-base-samba4.so'do_lock_fn+0xa4 [0xfffffd7fee70d534]
   #8 /usr/lib/samba/private/amd64/libsmbd-base-samba4.so'share_mode_do_locked_fn+0x86 [0xfffffd7fee7174b6]
   #9 /usr/lib/samba/amd64/libsmbconf.so.0'dbwrap_watched_do_locked_fn+0xfa [0xfffffd7fedf622ca]
   #10 /usr/lib/samba/private/amd64/libdbwrap-samba4.so'db_tdb_do_locked+0x12f [0xfffffd7feccb95cf]
   #11 /usr/lib/samba/private/amd64/libdbwrap-samba4.so'dbwrap_do_locked+0x48 [0xfffffd7feccb69a8]
   #12 /usr/lib/samba/amd64/libsmbconf.so.0'dbwrap_watched_do_locked+0x6f [0xfffffd7fedf60d7f]
   #13 /usr/lib/samba/private/amd64/libdbwrap-samba4.so'dbwrap_do_locked+0x48 [0xfffffd7feccb69a8]
   #14 /usr/lib/samba/private/amd64/libsmbd-base-samba4.so'share_mode_do_locked+0xd2 [0xfffffd7fee719b82]
   #15 /usr/lib/samba/private/amd64/libsmbd-base-samba4.so'do_lock+0xf0 [0xfffffd7fee70dfe0]
   #16 /usr/lib/samba/amd64/vfs/fruit.so'fruit_create_file+0x7ba [0xfffffd7fe88855aa]
   #17 /usr/lib/samba/private/amd64/libsmbd-base-samba4.so'smbd_smb2_request_process_create+0xa07 [0xfffffd7fee7d3237]
   #18 /usr/lib/samba/private/amd64/libsmbd-base-samba4.so'smbd_smb2_request_dispatch+0xc8f [0xfffffd7fee7c985f]
   #19 /usr/lib/samba/private/amd64/libsmbd-base-samba4.so'smbd_smb2_connection_handler+0x621 [0xfffffd7fee7ca7e1]
   #20 /usr/lib/samba/private/amd64/libtevent.so.0.10.2'tevent_common_invoke_fd_handler+0x80 [0xfffffd7fecd3a580]
   #21 /usr/lib/samba/private/amd64/libtevent.so.0.10.2'epoll_event_loop_once+0x22c [0xfffffd7fecd4180c]
   #22 /usr/lib/samba/private/amd64/libtevent.so.0.10.2'std_event_loop_once+0x40 [0xfffffd7fecd3f8f0]
   #23 /usr/lib/samba/private/amd64/libtevent.so.0.10.2'_tevent_loop_once+0x95 [0xfffffd7fecd39bd5]
   #24 /usr/lib/samba/private/amd64/libtevent.so.0.10.2'tevent_common_loop_wait+0x23 [0xfffffd7fecd39e43]
   #25 /usr/lib/samba/private/amd64/libtevent.so.0.10.2'std_event_loop_wait+0x40 [0xfffffd7fecd3f870]
   #26 /usr/lib/samba/private/amd64/libsmbd-base-samba4.so'smbd_process+0x777 [0xfffffd7fee7b8677]
   #27 /usr/lib/samba/sbin/amd64/smbd'smbd_accept_connection+0x189 [0x40d5b9]
   #28 /usr/lib/samba/private/amd64/libtevent.so.0.10.2'tevent_common_invoke_fd_handler+0x80 [0xfffffd7fecd3a580]
   #29 /usr/lib/samba/private/amd64/libtevent.so.0.10.2'epoll_event_loop_once+0x22c [0xfffffd7fecd4180c]
   #30 /usr/lib/samba/private/amd64/libtevent.so.0.10.2'std_event_loop_once+0x40 [0xfffffd7fecd3f8f0]
   #31 /usr/lib/samba/private/amd64/libtevent.so.0.10.2'_tevent_loop_once+0x95 [0xfffffd7fecd39bd5]
   #32 /usr/lib/samba/private/amd64/libtevent.so.0.10.2'tevent_common_loop_wait+0x23 [0xfffffd7fecd39e43]
   #33 /usr/lib/samba/private/amd64/libtevent.so.0.10.2'std_event_loop_wait+0x40 [0xfffffd7fecd3f870]
   #34 /usr/lib/samba/sbin/amd64/smbd'main+0x1a0f [0x40f9ff]
   #35 /usr/lib/samba/sbin/amd64/smbd'_start_crt+0x83 [0x408e73]
   #36 /usr/lib/samba/sbin/amd64/smbd'_start+0x18 [0x408dd8]

In this particular nested share_mode_do_locked() invocation, callback
comes through dbwrap_watched_do_locked_fn() where it fails to update
rec->value_valid which further gets assigned to static_share_mode_record
within share_mode_do_locked_fn().

BUG: https://bugzilla.samba.org/show_bug.cgi?id=14352

Signed-off-by: Anoop C S 
Reviewed-by: Ralph Boehme 
Reviewed-by: Volker Lendecke 

Autobuild-User(master): Ralph Böhme 
Autobuild-Date(master): Tue Apr 21 17:37:43 UTC 2020 on sn-devel-184

dbwrap: fix possible memleak and false result check.

2020-04-02T11:26:32+00:00

A cstatus != 0 or a data.dsize != sizeof(uint32_t)
does not guarantee to have no received data referenced by data.dptr.
Therefore, make sure data.dptr is free'd.
Reusing the same data structure as data input and data output parameter
can lead to wrong results, especially when the output parameters value
is used to detect errors. Create the additional local variable outdata
to prevent this issue.

Signed-off-by: Swen Schillig 
Reviewed-by: Volker Lendecke 
Reviewed-by: Martin Schwenke 

Autobuild-User(master): Martin Schwenke 
Autobuild-Date(master): Thu Apr  2 11:26:32 UTC 2020 on sn-devel-184

dbwrap: fix comment in code leading to wrong function parameter

2020-04-02T09:51:39+00:00

Signed-off-by: Swen Schillig 
Reviewed-by: Volker Lendecke 
Reviewed-by: Martin Schwenke