samba.git/python/samba/tests/source_chars.py, branch talloc-2.3.4

HEIMDAL: move code from source4/heimdal* to third_party/heimdal*

2022-01-19T21:41:59+00:00

This makes it clearer that we always want to do heimdal changes
via the lorikeet-heimdal repository.

Signed-off-by: Stefan Metzmacher 
Reviewed-by: Joseph Sutton 

Autobuild-User(master): Joseph Sutton 
Autobuild-Date(master): Wed Jan 19 21:41:59 UTC 2022 on sn-devel-184

tests: Update latin1 list and ignored file list for new Heimdal import

2022-01-19T20:50:35+00:00

NOTE: THIS COMMIT WON'T COMPILE/WORK ON ITS OWN!

Signed-off-by: Andrew Bartlett 
Reviewed-by: Stefan Metzmacher

pytest/source_char: check for mixed direction text

2021-12-03T18:53:43+00:00

As pointed out in https://lwn.net/Articles/875964, forbidding bidi
marker characters is not always going to be enough to avoid
right-to-left vs left-to-right confusion. Consider this:

$ python -c's = "b = x  # 2 * n * m"; print(s); print(s.replace("x", "א").replace("n", "ח"))'

b = x  # 2 * n * m
b = א  # 2 * ח * m

Those two lines are semantically the same, with the Hebrew letters
"א" and "ח" replacing "x" and "n". But they look like they mean
different things.

It is not enough to say we only allow these scripts (or indeed
non-ascii) in strings and comments, as demonstrated in this example:

$ python -c's = "b = \"x#\"  #  n"; print(s); print(s.replace("x", "א").replace("n", "ח"))'

b = "x#"  #  n
b = "א#"  #  ח

where the second line is visually disordered but looks valid. Any series
of neutral characters between teo RTL characters will be reversed (and
possibly mirrored).

In practice this affects one file, which is a text file for testing
unicode normalisation.

I think, for the reasons shown above, we are unlikely to see legitimate
RTL code outside perhaps of documentation files — but if we do, we can
add those files to the allow-list.

Signed-off-by: Douglas Bagnall 
Reviewed-by: Andrew Bartlett 

Autobuild-User(master): Andrew Bartlett 
Autobuild-Date(master): Fri Dec  3 18:53:43 UTC 2021 on sn-devel-184

pytest/source_chars: forget thirdparty/pep8 test file

2021-11-19T12:35:39+00:00

Signed-off-by: Douglas Bagnall 
Reviewed-by: Noel Power

lib/replace/timegm: use utf-8

2021-11-17T05:27:39+00:00

Signed-off-by: Douglas Bagnall 
Reviewed-by: Ralph Boehme 

Autobuild-User(master): Ralph Böhme 
Autobuild-Date(master): Wed Nov 17 05:27:39 UTC 2021 on sn-devel-184

s4/auth/gensec/gensec_krb5_heimdal: use utf-8

2021-11-17T04:36:37+00:00

Signed-off-by: Douglas Bagnall 
Reviewed-by: Ralph Boehme

test/blackbox/test_samba-tool_ntacl: use utf-8

2021-11-17T04:36:37+00:00

Signed-off-by: Douglas Bagnall 
Reviewed-by: Ralph Boehme

s3/modules/vfs_acl_common.h: use utf-8

2021-11-17T04:36:36+00:00

Signed-off-by: Douglas Bagnall 
Reviewed-by: Ralph Boehme

test/bad_chars: ensure our tests could fail

2021-11-17T04:36:36+00:00

Signed-off-by: Douglas Bagnall 
Reviewed-by: Ralph Boehme

pytests: check that we don't have bad format characters

2021-11-17T04:36:36+00:00

Unicode has format control characters that affect the appearance —
including the apparent order — of other characters. Some of these,
like the bidi controls (for mixing left-to-right scripts with
right-to-left scripts) can be used make text that means one thing look
very much like it means another thing.

The potential for duplicity using these characters has recently been
publicised under the name “Trojan Source”, and CVE-2021-42694. A
specific example, as it affects the Rust language is CVE-2021-42574.

We don't have many format control characters in our code — in fact,
just the non-breaking space (\u200b) and the redundant BOM thing
(\ufeff), and this test aims to ensure we keep it that way.

The test uses a series of allow-lists and deny-lists to check most
text files for unknown format control characters. The filtering is
fairly conservative but not exhaustive. For example, XML and text
files are checked, but UTF-16 files are not.

Signed-off-by: Douglas Bagnall 
Reviewed-by: Ralph Boehme