summaryrefslogtreecommitdiff
path: root/python/samba/emulate
AgeCommit message (Collapse)AuthorFilesLines
2020-10-02python2 reduction: Merge remaining compat code into commonDavid Mulder1-1/+1
The remaining compat code (get_string, get_bytes, cmp) are useful helper routines which we should simply merge into common (especially since there is some duplication here). Signed-off-by: David Mulder <dmulder@suse.com> Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz> Autobuild-User(master): David Mulder <dmulder@samba.org> Autobuild-Date(master): Fri Oct 2 14:49:36 UTC 2020 on sn-devel-184
2020-04-27traffic_packets: fix SyntaxWarning: "is" with a literalDavid Disseldorp1-1/+1
Python 3.8 adds this warning via https://bugs.python.org/issue34850: the "is" and "is not" operator sometimes is used with string and numerical literals. This code "works" on CPython by accident, because of caching on different levels (small integers and strings caches, interned strings, deduplicating constants at compile time). But it shouldn't work on other implementations, and can not work even on early or future CPython versions. Reported-by: L. van Belle <belle@samba.org> Signed-off-by: David Disseldorp <ddiss@samba.org> Reviewed-by: Noel Power <noel.power@suse.com> Autobuild-User(master): David Disseldorp <ddiss@samba.org> Autobuild-Date(master): Mon Apr 27 12:19:59 UTC 2020 on sn-devel-184
2019-09-24traffic.py: avoid inefficient string concatenationsBjörn Jacke1-3/+1
Signed-off-by: Bjoern Jacke <bjacke@samba.org> Reviewed-by: Andrew Bartlett <abartlet@samba.org>
2019-07-24traffic_replay: Avoid DB full scans in LDAP searchesTim Beale2-0/+63
When generating LDAP search traffic, a full DB scan can be very costly. Avoiding full-scan LDAP searches means that we can run traffic_replay against a 100K user DB and get some sane results. Because the traffic_learner doesn't record the LDAP search filter at all, the traffic_replay LDAP searches default to being full scans. Doing full scans meant that the LDAP search was usually the first packet type to exceed the max latency and fail the test. It could also skew results for the other packet types by creating big demands on memory/CPU/ DB-lock-time. It's hard to know for sure exactly what real-world LDAP searches will look like, but let's assume full scan searches will be fairly rare. In traffic-model files we've collected previously, some of the attributes are fairly unique (e.g. pKIExtendedKeyUsage), and as there are some LDAP queries specified in MS specs (such as MS-GPOL and MS-WCCE), it allows us to infer what the search filter might be. Signed-off-by: Tim Beale <timbeale@catalyst.net.nz> Reviewed-by: Andrew Bartlett <abartlet@samba.org>
2019-07-24traffic replay: Store the instance id in the replay contextGary Lockyer1-1/+3
Store the traffic runner instance id in the replay context. Will be used in subsequent commits. Signed-off-by: Gary Lockyer <gary@catalyst.net.nz> Reviewed-by: Andrew Bartlett <abartlet@samba.org>
2019-07-24traffic_replay: Make use of SCOPE_BASE explicitTim Beale1-2/+2
i.e. avoid hard-coded numbers. Signed-off-by: Tim Beale <timbeale@catalyst.net.nz> Reviewed-by: Andrew Bartlett <abartlet@samba.org>
2019-07-24traffic_replay: Store total conversations on the replay contextTim Beale1-0/+3
This is useful info to know, and will be used in subsequent commits. Signed-off-by: Tim Beale <timbeale@catalyst.net.nz> Reviewed-by: Andrew Bartlett <abartlet@samba.org>
2019-05-01traffic: load dns query from file and write stats to fileJoe Guo1-13/+69
Signed-off-by: Joe Guo <joeg@catalyst.net.nz> Reviewed-by: Andrew Bartlett <abartlet@samba.org> Reviewed-by: Andreas Schneider <asn@samba.org> Autobuild-User(master): Andrew Bartlett <abartlet@samba.org> Autobuild-Date(master): Wed May 1 01:10:42 UTC 2019 on sn-devel-184
2019-04-30traffic: define kerberos_state to simplify codeJoe Guo1-25/+10
Signed-off-by: Joe Guo <joeg@catalyst.net.nz> Reviewed-by: Andrew Bartlett <abartlet@samba.org> Reviewed-by: Andreas Schneider <asn@samba.org>
2019-04-30traffic: make code more pythonicJoe Guo1-9/+3
Signed-off-by: Joe Guo <joeg@catalyst.net.nz> Reviewed-by: Andrew Bartlett <abartlet@samba.org> Reviewed-by: Andreas Schneider <asn@samba.org>
2019-01-08traffic_replay: use packets per second as primary scaleDouglas Bagnall1-6/+14
The old -S/--scale-traffic is relative to the original model, which made its relationship to true traffic volumes quite opaque Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz> Reviewed-by: Andrew Bartlett <abartlet@samba.org>
2019-01-08traffic: add option to reanimate dying conversationsDouglas Bagnall1-4/+20
The traffic model is generated from a window in time, which makes conversations appear to start and stop unnaturally at the window boundaries. When the window is short compared to the traffic replay time and the true expected conversation length, this has a significant distorting effect, leading to more conversations than would be expected to generate a given number of packets. To offset this slightly we add the --conversation-persistence option which tries to convert apparent death into a longish wait. Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz> Reviewed-by: Andrew Bartlett <abartlet@samba.org>
2019-01-08traffic: Conversation.add_short_packet is discerning about packetsDouglas Bagnall1-1/+4
If the packets really wouldn't do anything, we might as well not add them. Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz> Reviewed-by: Andrew Bartlett <abartlet@samba.org>
2019-01-08traffic: remove useless branch in stats reportDouglas Bagnall1-25/+12
This completes the work of 68c64c634ad7576b2ab5ef838c203320afd90f44, but differs from that in that it makes no actual change because isatty was not being called so was always evaluated as true. Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz> Reviewed-by: Andrew Bartlett <abartlet@samba.org>
2019-01-08traffic: generate more statistics in replayDouglas Bagnall1-30/+69
Add more "header" values indicating the progress of the run as a whole. The new fields are: Max sleep miss - the longest sleep() oversleep. Indicates client load. Maximum lag - the longest gap between a planned packet time and its actual time. Start lag - the longest gap between intended and actual conversation start. Planned conversations - how many conversations we meant to have. Planned packets - how many "packets" we thought we were making. Not all "packets" result in actual operations or packets. Unfinished conversations - how many conversations had not finished when they were killed. Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz> Reviewed-by: Andrew Bartlett <abartlet@samba.org>
2019-01-08traffic: avoid bare except: clausesDouglas Bagnall1-2/+2
Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz> Reviewed-by: Andrew Bartlett <abartlet@samba.org>
2019-01-08traffic: simplify tests for badpassword_frequencyDouglas Bagnall1-3/+4
x <= 0 will fail one or both of the other test clauses. Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz> Reviewed-by: Andrew Bartlett <abartlet@samba.org>
2019-01-08traffic: assign context domain at startDouglas Bagnall1-7/+1
Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz> Reviewed-by: Andrew Bartlett <abartlet@samba.org>
2019-01-08traffic: traffic_replay --latency-timeout to control final waitDouglas Bagnall1-1/+2
Conversations that haven't finished within some acceptable margin of on-time can be said to have failed. This is where you specify that margin. Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz> Reviewed-by: Andrew Bartlett <abartlet@samba.org>
2019-01-08traffic_replay: --stop-on-any-error option to not ignore client troubleDouglas Bagnall1-0/+3
Sometimes you want to know if any client is crashing for any reason. In those times use --stop-on-any-error for an early exit. Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz> Reviewed-by: Andrew Bartlett <abartlet@samba.org>
2019-01-08traffic: evoke OS error codes by name, not numberDouglas Bagnall1-2/+3
Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz> Reviewed-by: Andrew Bartlett <abartlet@samba.org>
2019-01-08traffic: reinitialise RNG in each childDouglas Bagnall1-2/+8
Otherwise they all replay using the same random sequence. Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz> Reviewed-by: Andrew Bartlett <abartlet@samba.org>
2019-01-08traffic: avoid generating conversations without packetsDouglas Bagnall1-0/+28
Some "packets" don't generate any actual traffic. If we have a conversation consisting only of those, we can avoid forking a client for it. This *slightly* increases the load over that which would be generated otherwise for a given traffic rate, but that's OK. Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz> Reviewed-by: Andrew Bartlett <abartlet@samba.org>
2019-01-08traffic: rework conversation generation to better use memoryDouglas Bagnall1-149/+173
Use less memory altogether and don't allocated shared mutable before the fork. Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz> Reviewed-by: Andrew Bartlett <abartlet@samba.org>
2019-01-08traffic: generate sparser descriptions of conversationsDouglas Bagnall1-4/+3
Rather than building all the packets at this point, we stick to the barest details of the packets (which is all the model gives us anyway). The advantage is that will take a lot less memory, which matters because this process forks into many clients that were sharing and mutate the conversation list. Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz> Reviewed-by: Andrew Bartlett <abartlet@samba.org>
2019-01-08traffic: new version of model with packet_rate, version numberDouglas Bagnall1-46/+72
Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz> Reviewed-by: Andrew Bartlett <abartlet@samba.org>
2019-01-08traffic: fix hash non-determinism when loading JSONDouglas Bagnall1-0/+2
Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz> Reviewed-by: Tim Beale <timbeale@samba.org>
2019-01-08traffic: Conversation gets given id, list of short packetsDouglas Bagnall1-8/+8
Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz> Reviewed-by: Andrew Bartlett <abartlet@samba.org>
2019-01-08traffic: rename packet_rate -> replay_speed for accuracy and roomDouglas Bagnall1-5/+6
We are soon going to have a self.packet_rate, and replay_speed is more accurate in this case. Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz> Reviewed-by: Andrew Bartlett <abartlet@samba.org>
2019-01-08traffic: initialise conversation specific lists after the forkDouglas Bagnall1-12/+12
Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz> Reviewed-by: Andrew Bartlett <abartlet@samba.org>
2019-01-08traffic: split is_a_real_packet() function out of classDouglas Bagnall1-17/+21
So we can use it to determine whether a packet should be a Packet before making the leap. Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz> Reviewed-by: Andrew Bartlett <abartlet@samba.org>
2019-01-08traffic: give Packet __slots__ for reduced memoryDouglas Bagnall1-1/+10
Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz> Reviewed-by: Andrew Bartlett <abartlet@samba.org>
2019-01-08traffic: make random_colour_print more deterministic, silent on demandDouglas Bagnall1-9/+21
Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz> Reviewed-by: Andrew Bartlett <abartlet@samba.org>
2019-01-08traffic: use namedtuple for ConversationAccounts namespace classDouglas Bagnall1-9/+10
Saving memory, which reduces fork overhead. Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz> Reviewed-by: Andrew Bartlett <abartlet@samba.org>
2019-01-08traffic: fix mk_masked_dir doc and commentsDouglas Bagnall1-1/+2
Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz> Reviewed-by: Andrew Bartlett <abartlet@samba.org>
2019-01-08traffic: Packet.from_line classmethod uses cls varDouglas Bagnall1-3/+3
Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz> Reviewed-by: Andrew Bartlett <abartlet@samba.org>
2018-12-11various: Remove references to about to be deleted thirdparty/dnspythonNoel Power1-1/+0
Signed-off-by: Noel Power <noel.power@suse.com> Reviewed-by: Andrew Bartlett <abartlet@samba.org>
2018-12-04traffic_replay: Add a max-members option to cap group sizeTim Beale1-4/+30
traffic_replay tries to distribute the users among the groups in a realistic manner - some groups will have almost all users in them. However, this becomes a problem when testing a really large database, e.g. we may want 100K users, but no more than 5K users in each group. This patch adds a max-member option so we can limit how big the groups actually get. If we detect that a group exceeds the max-members, we reset the group's probability (of getting selected) to zero, and then recalculate the cumulative distribution. The means that the group should no longer get selected by generate_random_membership(). (Note we can't completely remove the group from the list because that changes the list-index-to-group-ID mapping). Signed-off-by: Tim Beale <timbeale@catalyst.net.nz> Reviewed-by: Andrew Bartlett <abartlet@samba.org> Autobuild-User(master): Andrew Bartlett <abartlet@samba.org> Autobuild-Date(master): Tue Dec 4 12:22:50 CET 2018 on sn-devel-144
2018-12-04traffic: Rework how assignments are generated slightlyTim Beale1-19/+14
We want to cap the number of members that can be in a group. But first, we need to tweak how the assignment dict gets generated, so that we get rid of the intermediary set. Signed-off-by: Tim Beale <timbeale@catalyst.net.nz> Reviewed-by: Andrew Bartlett <abartlet@samba.org>
2018-11-07traffic_replay: Rework machine accounts to remove redundant codeTim Beale1-55/+5
generate_users_and_groups() now generates the machine acounts as well as the user accounts, so it seems there's no need to also have generate_traffic_accounts(), which does the same job. Instead, we can just pass through the number of machine acounts to generate_users_and_groups() and delete the other function. Also updated generate_users_and_groups() so that machine_accounts is no longer optional (we want to create machine accounts in all cases). Signed-off-by: Tim Beale <timbeale@catalyst.net.nz> Reviewed-by: Andrew Bartlett <abartlet@samba.org>
2018-11-07traffic_replay: Make packet generation work on a pre-populated DB againTim Beale1-3/+13
Generate separate machine accounts for populating a large DB vs replaying network traffic. We want to use different userAccountControl flags in each of the above cases (i.e. commit 3338a3e257fa9f28). However, this means that once you use the --generate-users-only option, you can't replay network packets against the machine accounts. We can avoid this problem by creating separate machine accounts for each of 2 different cases, e.g. STGM-0-x machines for traffic-replay, and PC-0-x machines for padding out the database. Signed-off-by: Tim Beale <timbeale@catalyst.net.nz> Reviewed-by: Andrew Bartlett <abartlet@samba.org>
2018-11-07traffic_replay: Make sure naming assumptions are in a single placeTim Beale1-7/+11
The traffic_replay group/user/machine account names follow a standard format. This adds a function to generate the machine-name. It also makes sure the existing user_name() function gets called in all applicable places. Signed-off-by: Tim Beale <timbeale@catalyst.net.nz> Reviewed-by: Andrew Bartlett <abartlet@samba.org>
2018-11-07traffic_replay: Move 'traffic account' flag up a levelTim Beale1-4/+7
We create machine accounts for 2 different purposes: 1). For traffic generation, i.e. testing realistic network packets. 2). For generating a realistic large DB. Unfortunately, we want to use different userAccountControl flags for the 2 different cases. Commit 3338a3e257fa9f28 changed the flags used for case #2, but this breaks case #1. The problem is generate_users_and_groups() is called in both cases, so we want the 'traffic account' flag passed into that function. This ensures that the machine accounts get created with the appropriate userAccountControl flags for the particular case you want to test. Signed-off-by: Tim Beale <timbeale@catalyst.net.nz> Reviewed-by: Andrew Bartlett <abartlet@samba.org>
2018-11-07traffic_replay: Move machine account creationTim Beale1-7/+7
I was assuming that generate_users_and_groups() only gets called in the --generate-users-only case. However, it also gets called in the default traffic replay case. This patch reworks the code so that the number of machine accounts to create gets passed in, and the 'create 25% more computers than users' assumption only applies to the --generate-users-only case. Signed-off-by: Tim Beale <timbeale@catalyst.net.nz> Reviewed-by: Andrew Bartlett <abartlet@samba.org>
2018-11-05traffic: Machine accounts were generated as critical objectsTim Beale1-5/+16
Due to the userAccountControl flags we were specifying, the machine accounts were all created as critical objects. When trying to populate 1000s of machine accounts in a DB, this makes replication unnecessarily slow (because it has to replicate them all twice). This patch changes it so when we're just creating machine accounts for the purpose of populating a semi-realistic DB, we jsut use the default WORKSTATION_TRUST_ACCOUNT flag. Note that for the accounts used for traffic-replay, we apparently need the existing flags in order for the DC to accept certain requests. Signed-off-by: Tim Beale <timbeale@catalyst.net.nz> Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz> Autobuild-User(master): Tim Beale <timbeale@samba.org> Autobuild-Date(master): Mon Nov 5 03:43:24 CET 2018 on sn-devel-144
2018-11-04traffic_replay: Generate machine accounts as well as usersTim Beale1-2/+25
Currently the tool only generates the machine accounts needed for traffic generation. However, this isn't realistic if we're trying to use the tool to generate users to simulate a large network. This patch generates machine accoutns along with the user accounts. Note we assume there will be more computer accounts than users in a real network (e.g. work laptops, servers, etc), so generate slightly more computer accounts. Signed-off-by: Tim Beale <timbeale@catalyst.net.nz> Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
2018-11-04traffic_replay: Improve user generation debugTim Beale1-17/+17
When creating 1000s of users you currently get a lot of debug, but at the same time you have no idea how far through creating the users you actually are. Instead of logging every single user account that's created, log every 50th (as well as how far through the overall generation we are). Logger already includes timestamps, so we can remove generating the timestamp diff manually. User creation is the slowest operation - adding groups/memberships is much faster, so we don't need to log as frequently. Note that there is a usability trade-off on how frequently we log depending on whether the user is using the slower (but more common) method of going via LDAP, vs the much faster (but more obscure) method of writing directly to sam.ldb with ldb:nosync=true. In my tests, we end up logging every ~30-ish secs with LDAP, and every ~3 seconds with direct file writes. Signed-off-by: Tim Beale <timbeale@catalyst.net.nz> Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
2018-11-04traffic_replay: Convert print() to logger.info()Tim Beale1-13/+9
Using logger is more helpful here because it includes timestamps, so we can see how long things are taking. It's also more consistent with the rest of the traffic_replay logging. Signed-off-by: Tim Beale <timbeale@catalyst.net.nz> Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
2018-11-04traffic_replay: Write group memberships once per groupTim Beale1-13/+30
Each user-group membership was being written to the DB in a single operation. With large numbers of users (e.g. 10,000 in average 15 groups each), this becomes a lot of operations (e.g. 150,000). This patch reworks the code so that we write the memberships for a group in one operation. E.g. instead of 150,000 DB operations, we might make 1,500. This makes writing the group memberships several times faster. Note that rthere is a performance vs memory tradeoff. When we hit 10,000+ members in a group, memory-usage in the underlying DB modify operation becomes very inefficient/costly. So we avoid potential memory usage problems by writing no more than 1,000 users to a group at once. Signed-off-by: Tim Beale <timbeale@catalyst.net.nz> Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
2018-11-04traffic_replay: Re-organize assignments to be group-basedTim Beale1-19/+32
We can speed up writing the group memberships by adding multiple users to a group in a single DB modify operation. To do this, we first need to reorganize the assignments so instead of being a set of tuples, it's a dictionary where key=group and value=list-of-users-in-group. add_users_to_groups() now iterates through the users/groups slightly differently, but mostly it's just indentation changes. We haven't changed the number of DB operations yet - we'll do that in the next patch. Signed-off-by: Tim Beale <timbeale@catalyst.net.nz> Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
generated by cgit v1.2.3 (git 2.25.1) at 2026-05-07 05:47:04 +0000