Compare commits

...
Author SHA1 Message Date
John Sully c801414b40 Perf: remove mfence 2020-10-04 03:15:18 +00:00
John Sully f4c317d285 Initial implementation of replication-db-disable config 2020-10-03 23:57:35 +00:00
John Sully 420ccdfbae Fix module test failures due to locking 2020-10-01 00:08:54 +00:00
John Sully 41466a4147 Fix TLS test failure due to bad merge 2020-09-30 20:21:40 +00:00
John Sully 9798f2d711 Keep redis-cli.c C89 compatible 2020-09-30 20:17:51 +00:00
John Sully 5f3543921b Mac build break fix 2020-09-30 20:17:02 +00:00
John Sully a41366cc3a Fix stream replication failure with active replication, issue #238 2020-09-30 20:12:54 +00:00
John Sully 4c7e4b91a6 Merge tag '6.0.8' into unstable 2020-09-30 19:47:55 +00:00
John Sully 28183f4b66 Test RDB merge on load with active replication 2020-09-29 03:26:06 +00:00
John Sully a0abc1eddd Drop min-clients-per-thread to a more reasonable number 2020-09-28 19:17:36 +00:00
John Sully 676644f2db Support read only replicas attaching to active replicas (Bug #229) 2020-09-23 04:16:08 +00:00
John Sully 22ac56d62d Active replicas memory leak 2020-09-22 21:38:35 +00:00
John Sully 143f7f634d Merge branch 'unstable' of https://github.com/JohnSully/KeyDB into unstable 2020-09-21 04:42:46 +00:00
Dmitry Dorofeev 955faef23a Fix to support FreeBSD pthread_getthreadid_np call 2020-09-21 00:41:41 -04:00
John Sully 2e159e6ea3 Fix TSAN errors in fastlock 2020-09-21 03:26:50 +00:00
John Sully 02b533fec3 Fix issue #222 2020-09-21 01:09:58 +00:00
John Sully de3099e6d8 Make multimaster replication info statelessly paresable 2020-09-20 23:57:06 +00:00
John Sully 368f67f422 Dramatically improve the performance of subkey expires 2020-09-20 23:30:21 +00:00
John Sully 13c9a094d1 Reduce test noise 2020-09-19 22:00:17 +00:00
John Sully 0f8b7a19ef Fix CI failure due to apt host changes 2020-09-19 21:49:59 +00:00
John Sully 8f7d977f90 Avoid memory overhead in situations it won't be used (active rep only) 2020-09-19 21:44:59 +00:00
John Sully 064440ee84 Update endurance test for ubuntu 20 2020-09-17 02:10:18 +00:00
John Sully 50641d0569 Fix build break 2020-09-17 00:53:56 +00:00
John Sully f5ecc77e29 master_global_link_status should respect the quorum 2020-09-17 00:31:10 +00:00
John Sully 655bb6e21c Allow reads during an active replica load, optionally allow writes as an experimental feature 2020-09-17 00:13:03 +00:00
John Sully ab4cdd2ed4 Implement replica-quorum config 2020-09-16 03:26:31 +00:00
Yossi Gottlieb d3834c5069 TLS: Session caching configuration support. (#7420)
* TLS: Session caching configuration support.
* TLS: Remove redundant config initialization.
2020-09-16 02:10:29 +00:00
John Sully 7a1b2c58c3 Protocol error in client command 2020-09-11 20:48:18 +00:00
John Sully 7fc6d6c6b1 Fix more protocol errors 2020-09-11 20:47:27 +00:00
John Sully dec5db4e71 Fix protocol difference with Redis 2020-09-11 20:08:26 +00:00
Oran Agra 03b59cd5f0 6.0.8 2020-09-10 14:09:00 +03:00
Oran Agra 5380716a70 Fix RESP3 response for HKEYS/HVALS on non-existing key 2020-09-10 14:09:00 +03:00
Oran Agra 18b30467f7 Fix leak in new blockedclient module API test
(cherry picked from commit 0e20ad14a6a857cb168b808f94721df19b23dc0c)
2020-09-10 14:09:00 +03:00
Yossi Gottlieb 0052d5196d Tests: fix oom-score-adj false positives. (#7772)
The key save delay is too short and on certain systems the child process
is gone before we have a chance to inspect it.

(cherry picked from commit b2a73c404b)
2020-09-10 14:09:00 +03:00
杨博东 b55b0ea0f2 Tests: Add aclfile load and save tests (#7765)
improves test coverage

(cherry picked from commit 0666267d27)
2020-09-10 14:09:00 +03:00
Roi Lipman 25e211bcf0 RM_ThreadSafeContextTryLock a non-blocking method for acquiring GIL (#7738)
Co-authored-by: Yossi Gottlieb <yossigo@gmail.com>
Co-authored-by: Oran Agra <oran@redislabs.com>
(cherry picked from commit 042189fd87)
2020-09-10 14:09:00 +03:00
Yossi Gottlieb 635704d424 Tests: validate CONFIG REWRITE for all params. (#7764)
This is a catch-all test to confirm that that rewrite produces a valid
output for all parameters and that this process does not introduce
undesired configuration changes.

(cherry picked from commit a8b7268911)
2020-09-10 14:09:00 +03:00
Oran Agra 47364bce69 Change THP warning to use madvise rather than never (#7771)
completes b2419c31c1

(cherry picked from commit 1461f02deb)
2020-09-10 14:09:00 +03:00
Itamar Haber 3cd6c26056 Documents RM_Call's fmt (#5448)
Improve RM_Call inline documentation about the fmt argument
so that we don't completely depend on the web docs.

Co-authored-by: Oran Agra <oran@redislabs.com>
(cherry picked from commit ce15620dc1)
2020-09-10 14:09:00 +03:00
Jan-Erik Rediger 8b0d797ecd Check that THP is not set to always (madvise is ok) (#4001)
THP can also be set to madvise, in which case it shouldn't cause
problems for Redis since redis (or the allocator) doesn't use madvise
to activate it.

(cherry picked from commit b2419c31c1)
2020-09-10 14:09:00 +03:00
Yossi Gottlieb 3c8b394511 Tests: clean up stale .cli files. (#7768)
(cherry picked from commit 918abd7276)
2020-09-10 14:09:00 +03:00
Eran Liberty 84ba2ec82b Allow exec with read commands on readonly replica in cluster (#7766)
There was a bug. Although cluster replicas would allow read commands,
they would not allow a MULTI-EXEC that's composed solely of read commands.
Adds tests for coverage.

Co-authored-by: Oran Agra <oran@redislabs.com>
Co-authored-by: Eran Liberty <eranl@amazon.com>
(cherry picked from commit b120366d48)
2020-09-10 14:09:00 +03:00
Yossi Gottlieb cf3375bf7c Fix CONFIG REWRITE of oom-score-adj-values. (#7761)
(cherry picked from commit 750acf3a45)
2020-09-10 14:09:00 +03:00
Oran Agra 643d3e0b79 handle cur_test for nested tests
if there are nested tests and nested servers, we need to restore the
previous value of cur_test when a test exist.

example:
```
test{test 1} {
	start_server {
		test{test 1.1 - master only} {
		}
		start_server {
		    test{test 1.2 - with replication} {
            }
		}
	}
}
```
when `test 1.1 - master only exists`, we're still inside `test 1`

(cherry picked from commit 0a1e734193)
2020-09-10 14:09:00 +03:00
Oran Agra 874c3a9978 Add daily CI for MacOS (#7759)
(cherry picked from commit 5496b4a7cd)
2020-09-10 14:09:00 +03:00
bodong.ybd 781e50d41f Tests: Some fixes for macOS
1) cur_test: when restart_server, "no such variable" error occurs
  ./runtest --single integration/rdb
  test {client freed during loading}
      SET ::cur_test
      restart_server
        kill_server
          test "Check for memory leaks (pid $pid)"
          SET ::cur_test
          UNSET ::cur_test
      UNSET ::cur_test // This global variable has been unset.

2) `ps --ppid` not available on macOS platform, can be replaced with
`pgrep -P pid`.

(cherry picked from commit f22fa9594d)
2020-09-10 14:09:00 +03:00
Oran Agra 57dbd6e90d Fix cluster consistency-check test (#7754)
This test was failing from time to time see discussion at the bottom of #7635
This was probably due to timing, the DEBUG SLEEP executed by redis-cli
didn't sleep for enough time.

This commit changes:
1) use SET-ACTIVE-EXPIRE instead of DEBUG SLEEP
2) reduce many `after` sleeps with retry loops to speed up the test.
3) add many comment explaining the different steps of the test and
   it's purpose.
4) config appendonly before populating the volatile keys, so that they'll
   be part of the AOF command stream rather than the preamble RDB portion.

other complications: recently kill_instance switched from SIGKILL to
SIGTERM, and this would sometimes fail since there was an AOFRW running
in the background. now we wait for it to end before attempting the kill.

(cherry picked from commit b491d477c3)
2020-09-10 14:09:00 +03:00
Yossi Gottlieb e9fef49e12 Tests: fix unmonitored servers. (#7756)
There is an inherent race condition in port allocation for spawned
servers. If a server fails to start because a port is taken, a new port
is allocated. This fixes a problem where the logs are not truncated and
as a result a large number of unmonitored servers are started.

(cherry picked from commit 2df4cb93ac)
2020-09-10 14:09:00 +03:00
Oran Agra a227bc8053 fix broken cluster/sentinel tests by recent commit (#7752)
2b998de46 added a file for stderr to keep valgrind log but i forgot to
add a similar thing when valgrind isn't being used.
the result is that `glob */err.txt` fails.

(cherry picked from commit 42ba7a1b75)
2020-09-10 14:09:00 +03:00
Oran Agra d410dc3162 Improve valgrind support for cluster tests (#7725)
- redirect valgrind reports to a dedicated file rather than console
- try to avoid killing instances with SIGKILL so that we get the memory
  leak report (killing with SIGTERM before resorting to SIGKILL)
- search for valgrind reports when done, print them and fail the tests
- add --dont-clean option to keep the logs on exit
- fix exit error code when crash is found (would have exited with 0)

changes that affect the normal redis test suite:
- refactor check_valgrind_errors into two functions one to search and
  one to report
- move the search half into util.tcl to serve the cluster tests too
- ignore "address range perms" valgrind warnings which seem non relevant.

(cherry picked from commit 2b998de460)
2020-09-10 14:09:00 +03:00
Oran Agra 41c7c7919c test infra - add durable mode to work around test suite crashing
in some cases a command that returns an error possibly due to a timing
issue causes the tcl code to crash and thus prevents the rest of the
tests from running. this adds an option to make the test proceed despite
the crash.
maybe it should be the default mode some day.

(cherry picked from commit fe5da2e60d)
2020-09-10 14:09:00 +03:00
Oran Agra db6c763d8b test infra - wait_done_loading
reduce code duplication in aof.tcl.
move creation of clients into the test so that it can be skipped

(cherry picked from commit 1b7ba44e79)
2020-09-10 14:09:00 +03:00
Oran Agra 72d6f966ac test infra - flushall between tests in external mode
(cherry picked from commit b65e5aca86)
2020-09-10 14:09:00 +03:00
Oran Agra 28e074608c test infra - improve test skipping ability
- skip full units
- skip a single test (not just a list of tests)
- when skipping tag, skip spinning up servers, not just the tests
- skip tags when running against an external server too
- allow using multiple tags (split them)

(cherry picked from commit 677d14c213)
2020-09-10 14:09:00 +03:00
Oran Agra 5b8de5b7f2 test infra - reduce disk space usage
this is important when running a test with --loop

(cherry picked from commit e3e69c25fd)
2020-09-10 14:09:00 +03:00
Oran Agra bce350c666 test infra - write test name to logfile
(cherry picked from commit 9d527d076b)
2020-09-10 14:09:00 +03:00
Yossi Gottlieb 332aa2f919 redis-cli: fix writeConn() buffer handling. (#7749)
Fix issues with writeConn() which resulted with corruption of the stream by leaving an extra byte in the buffer. The trigger for this is partial writes or write errors which were not experienced on Linux but reported on macOS.

(cherry picked from commit 58e5feb3f4)
2020-09-10 14:09:00 +03:00
Oran Agra e4db67ea57 Print server startup messages after daemonization (#7743)
When redis isn't configured to have a log file, having these prints
before damonization puts them in the calling process stdout rather than
/dev/null

(cherry picked from commit 8b0747d657)
2020-09-10 14:09:00 +03:00
Thandayuthapani b983015dd4 Add masters/replicas options to redis-cli --cluster call command (#6491)
* Add master/slave option in --cluster call command

* Update src/redis-cli.c

* Update src/redis-cli.c

Co-authored-by: Itamar Haber <itamar@redislabs.com>
(cherry picked from commit f22f64f0db)
2020-09-10 14:09:00 +03:00
Oran Agra 9b718d41f0 fix README about BUILD_WITH_SYSTEMD usage (#7739)
BUILD_WITH_SYSTEMD is an internal variable. Users should use USE_SYSTEMD=yes.

(cherry picked from commit 747b4004ea)
2020-09-10 14:09:00 +03:00
Yossi Gottlieb da33a18588 Fix double-make issue with make && make install. (#7734)
All user-supplied variables that affect the build should be explicitly
persisted.

Fixes #7254

(cherry picked from commit b35d6e5cff)
2020-09-10 14:09:00 +03:00
John Sully d47aeb1fc8 Fix whitespace 2020-09-07 03:35:46 +00:00
John Sully e47584b286 Dramatically improve perf by blocking commands 2020-09-07 00:49:53 +00:00
Oran Agra dbea5f7a8d Redis 6.0.7 2020-09-01 09:27:58 +03:00
Oran Agra 4bb40a9688 Reduce the probability of failure when start redis in runtest-cluster #7554 (#7635)
When runtest-cluster, at first, we need to create a cluster use spawn_instance,
a port which is not used is choosen, however sometimes we can't run server on
the port. possibley due to a race with another process taking it first.
such as redis/redis/runs/896537490. It may be due to the machine problem or
In order to reduce the probability of failure when start redis in
runtest-cluster, we attemp to use another port when find server do not start up.

Co-authored-by: Oran Agra <oran@redislabs.com>
Co-authored-by: yanhui13 <yanhui13@meituan.com>
(cherry picked from commit e2d64485b8)
2020-09-01 09:27:58 +03:00
Yossi Gottlieb d2532d1335 Backport Lua 5.2.2 stack overflow fix. (#7733)
This fixes the issue described in CVE-2014-5461. At this time we cannot
confirm that the original issue has a real impact on Redis, but it is
included as an extra safety measure.

(cherry picked from commit d75ad774a9)
2020-09-01 09:27:58 +03:00
Leoš Literák 00d0d870d2 Update README.md with instructions how to build with systemd support (#7730)
#7728 - update instructions for systemd support

(cherry picked from commit 571571ca19)
2020-09-01 09:27:58 +03:00
Yossi Gottlieb f38e2802b6 Fix oom-score-adj on older distros. (#7724)
Don't assume `ps` handles `-h` to display output without headers and
manually trim headers line from output.

(cherry picked from commit b61b663895)
2020-09-01 09:27:58 +03:00
Wang Yuan 1b100a167a Fix wrong format specifiers of 'sdscatfmt' for the INFO command (#7706)
unlike printf, sdscatfmt doesn't take %d

(cherry picked from commit 43af28f5b4)
2020-09-01 09:27:58 +03:00
Wen Hui d4e1c88052 fix make warnings (#7692)
(cherry picked from commit e61adc0d89)
2020-09-01 09:27:58 +03:00
Nathan Scott 2c0c5a809c Annotate module API functions in redismodule.h for use with -fno-common (#6900)
In order to keep the redismodule.h self-contained but still usable with
gcc v10 and later, annotate each API function tentative definition with
the __common__ attribute.  This avoids the 'multiple definition' errors
modules will otherwise see for all API functions at link time.

Further details at gcc.gnu.org/gcc-10/porting_to.html

Turn the existing __attribute__ ((unused)), ((__common__)) and ((print))
annotations into conditional macros for any compilers not accepting this
syntax.  These macros only expand to API annotations under gcc.

Provide a pre- and post- macro for every API function, so that they can
be defined differently by the file that includes redismodule.h.

Removing REDISMODULE_API_FUNC in the interest of keeping the function
declarations readable.

Co-authored-by: Yossi Gottlieb <yossigo@gmail.com>
Co-authored-by: Oran Agra <oran@redislabs.com>
(cherry picked from commit 11cd983d58)
2020-09-01 09:27:58 +03:00
Yossi Gottlieb f6d04d01b9 Add oom-score-adj configuration option to control Linux OOM killer. (#1690)
Add Linux kernel OOM killer control option.

This adds the ability to control the Linux OOM killer oom_score_adj
parameter for all Redis processes, depending on the process role (i.e.
master, replica, background child).

A oom-score-adj global boolean flag control this feature. In addition,
specific values can be configured using oom-score-adj-values if
additional tuning is required.

(cherry picked from commit 2530dc0ebd)
2020-09-01 09:27:58 +03:00
Meir Shpilraien (Spielrein) 2fc915f509 see #7544, added RedisModule_HoldString api. (#7577)
Added RedisModule_HoldString that either returns a
shallow copy of the given String (by increasing
the String ref count) or a new deep copy of String
in case its not possible to get a shallow copy.

Co-authored-by: Itamar Haber <itamar@redislabs.com>
(cherry picked from commit 3f494cc49d)
2020-09-01 09:27:58 +03:00
ShooterIT e3cfc92180 [Redis-benchmark] Remove zrem test, add zpopmin test
(cherry picked from commit 6a06a5a597)
2020-09-01 09:27:58 +03:00
ShooterIT 7fcf29e615 [Redis-benchmark] Support zset type
(cherry picked from commit e5a50ed3c4)
2020-09-01 09:27:58 +03:00
Arun Ranganathan e81bac32fd Show threading configuration in INFO output (#7446)
Co-authored-by: Oran Agra <oran@redislabs.com>
(cherry picked from commit f6cad30bb6)
2020-09-01 09:27:58 +03:00
Meir Shpilraien (Spielrein) 2257f38b68 This PR introduces a new loaded keyspace event (#7536)
Co-authored-by: Oran Agra <oran@redislabs.com>
Co-authored-by: Itamar Haber <itamar@redislabs.com>
(cherry picked from commit 8d82639319)
2020-09-01 09:27:58 +03:00
Oran Agra 2165a78d10 Fix rejectCommand trims newline in shared error objects, hung clients (#7714)
65a3307bc (released in 6.0.6) has a side effect, when processCommand
rejects a command with pre-made shared object error string, it trims the
newlines from the end of the string. if that string is later used with
addReply, the newline will be missing, breaking the protocol, and
leaving the client hung.

It seems that the only scenario which this happens is when replying with
-LOADING to some command, and later using that reply from the CONFIG
SET command (still during loading). this will result in hung client.

Refactoring the code in order to avoid trimming these newlines from
shared string objects, and do the newline trimming only in other cases
where it's needed.

Co-authored-by: Guy Benoish <guy.benoish@redislabs.com>
(cherry picked from commit 9fcd9e191e)
2020-09-01 09:27:58 +03:00
valentinogeron e6f6731c66 EXEC with only read commands should not be rejected when OOM (#7696)
If the server gets MULTI command followed by only read
commands, and right before it gets the EXEC it reaches OOM,
the client will get OOM response.

So, from now on, it will get OOM response only if there was
at least one command that was tagged with `use-memory` flag

(cherry picked from commit b7289e912c)
2020-09-01 09:27:58 +03:00
Itamar Haber 5a9a653f3e Expands lazyfree's effort estimate to include Streams (#5794)
Otherwise, it is treated as a single allocation and freed synchronously. The following logic is used for estimating the effort in constant-ish time complexity:

1. Check the number of nodes.
1. Add an allocation for each consumer group registered inside the stream.
1. Check the number of PELs in the first CG, and then add this count times the number of CGs.
1. Check the number of consumers in the first CG, and then add this count times the number of CGs.

(cherry picked from commit 5b0a06af48)
2020-09-01 09:27:58 +03:00
Yossi Gottlieb ff0846cbb7 Add language servers stuff, test/tls to gitignore. (#7698)
(cherry picked from commit 5449a2a8b5)
2020-09-01 09:27:58 +03:00
Valentino Geron 3e2ed279fd Assert that setDeferredAggregateLen isn't called with negative value
In case the redis is about to return broken reply we want to crash
with assert so that we are notified about the bug. see #7687.

(cherry picked from commit 8b428cf0f7)
2020-09-01 09:27:58 +03:00
Valentino Geron 19ef1f371d Fix LPOS command when RANK is greater than matches
When calling to LPOS command when RANK is higher than matches,
the return value is non valid response. For example:
```
LPUSH l a
:1
LPOS l b RANK 5 COUNT 10
*-4
```
It may break client-side parser.

Now, we count how many replies were replied in the array.
```
LPUSH l a
:1
LPOS l b RANK 5 COUNT 10
*0
```

(cherry picked from commit 9204a9b2c2)
2020-09-01 09:27:58 +03:00
Yossi Gottlieb 8d79702d8a Tests: fix redis-cli with remote hosts. (#7693)
(cherry picked from commit f80f3f492a)
2020-09-01 09:27:58 +03:00
huangzhw 6cfcf76e0b RedisModuleEvent_LoadingProgress always at 100% progress (#7685)
It was also using the wrong struct, but luckily RedisModuleFlushInfo and RedisModuleLoadingProgress
are identical.

(cherry picked from commit a3d4d7bf68)
2020-09-01 09:27:58 +03:00
guybe7 c9478a3b4e Modules: Invalidate saved_oparray after use (#7688)
We wanna avoid a chance of someone using the pointer in it after it'll be freed / realloced.

(cherry picked from commit 65c24bd3d4)
2020-09-01 09:27:58 +03:00
杨博东 113d5ae872 Fix flock cluster config may cause failure to restart after kill -9 (#7674)
After fork, the child process(redis-aof-rewrite) will get the fd opened
by the parent process(redis), when redis killed by kill -9, it will not
graceful exit(call prepareForShutdown()), so redis-aof-rewrite thread may still
alive, the fd(lock) will still be held by redis-aof-rewrite thread, and
redis restart will fail to get lock, means fail to start.

This issue was causing failures in the cluster tests in github actions.

Co-authored-by: Oran Agra <oran@redislabs.com>
(cherry picked from commit cbaf3c5bba)
2020-09-01 09:27:58 +03:00
Raghav Muddur 1bd75aafdb Update clusterMsgDataPublish to clusterMsgModule (#7682)
Correcting the variable to clusterMsgModule.

(cherry picked from commit 34c3be365a)
2020-09-01 09:27:58 +03:00
Madelyn Olson ba48e7e9cd Fixed hset error since it's shared with hmset (#7678)
(cherry picked from commit cbd9af8583)
2020-09-01 09:27:58 +03:00
guybe7 4e19358d65 PERSIST should signalModifiedKey (Like EXPIRE does) (#7671)
(cherry picked from commit b87c288016)
2020-09-01 09:27:58 +03:00
Oran Agra 43bc1ea065 OOM Crash log include size of allocation attempt. (#7670)
Since users often post just the crash log in github issues, the log
print that's above it is missing.
No reason not to include the size in the panic message itself.

(cherry picked from commit 0f741a9e2d)
2020-09-01 09:27:58 +03:00
Wen Hui 3842bfd899 [module] using predefined REDISMODULE_NO_EXPIRE in RM_GetExpire (#7669)
It was already defined in the API header and the documentation, but not used by the implementation.

(cherry picked from commit 93d87d6d4c)
2020-09-01 09:27:58 +03:00
Oran Agra 6dcd641681 Trim trailing spaces in error replies coming from rejectCommand (#7668)
65a3307bc9 added rejectCommand which takes an robj reply and passes it
through addReplyErrorSafe to addReplyErrorLength.
The robj contains newline at it's end, but addReplyErrorSafe converts it
to spaces, and passes it to addReplyErrorLength which adds the protocol
newlines.

The result was that most error replies (like OOM) had extra two trailing
spaces in them.

(cherry picked from commit cdd925b289)
2020-09-01 09:27:58 +03:00
Yossi Gottlieb 1180537589 Module API: fix missing RM_CLIENTINFO_FLAG_SSL. (#7666)
The `REDISMODULE_CLIENTINFO_FLAG_SSL` flag was already a part of the `RedisModuleClientInfo` structure but was not implemented.

(cherry picked from commit 64c360c515)
2020-09-01 09:27:58 +03:00
Yossi Gottlieb 2e7ad58918 TLS: relax verification on CONFIG SET. (#7665)
Avoid re-configuring (and validating) SSL/TLS configuration on `CONFIG
SET` when TLS is not actively enabled for incoming connections, cluster
bus or replication.

This fixes failures when tests run without `--tls` on binaries that were
built with TLS support.

An additional benefit is that it's now possible to perform a multi-step
configuration process while TLS is disabled. The new configuration will
be verified and applied only when TLS is effectively enabled.

(cherry picked from commit fb2a94af3f)
2020-09-01 09:27:58 +03:00
Madelyn Olson 39ffc3702f Fixed timer warning (#5953)
(cherry picked from commit 79c506ebf0)
2020-09-01 09:27:58 +03:00
Wagner Francisco Mezaroba 31415cb27a allow --pattern to be used along with --bigkeys (#3586)
Adds --pattern option to cli's --bigkeys, --hotkeys & --scan modes

(cherry picked from commit e2a71338eb)
2020-09-01 09:27:58 +03:00
zhaozhao.zz 840a9951b5 redis-benchmark: fix wrong random key for hset (#4895)
(cherry picked from commit ff1e4a7063)
2020-09-01 09:27:58 +03:00
zhaozhao.zz 3c87d572b9 CLIENT_MASTER should ignore server.proto_max_bulk_len
(cherry picked from commit 589e610ebc)
2020-09-01 09:27:58 +03:00
zhaozhao.zz d3831559cf config: proto-max-bulk-len must be 1mb or greater
(cherry picked from commit bd4b33d7a2)
2020-09-01 09:27:58 +03:00
zhaozhao.zz 44ef19743e using proto-max-bulk-len in checkStringLength for SETRANGE and APPEND
(cherry picked from commit 2e69bfe44d)
2020-09-01 09:27:58 +03:00
YoongHM 171f83306e Start redis after network is online (#7639)
The two lines allow systemd to start redis.service after the network is online. Only after the network is online that Redis could bind to IP address other than 127.0.0.1 during initial boot up process.

(cherry picked from commit 8e937ce4cc)
2020-09-01 09:27:58 +03:00
Yossi Gottlieb 0314429c8f Run daily workflow on main repo only (no forks). (#7646)
(cherry picked from commit 944cca98c8)
2020-09-01 09:27:58 +03:00
WuYunlong 086707315f see #7250, fix signature of RedisModule_DeauthenticateAndCloseClient (#7645)
In redismodule.h, RedisModule_DeauthenticateAndCloseClient returns void
`void REDISMODULE_API_FUNC(RedisModule_DeauthenticateAndCloseClient)(RedisModuleCtx *ctx, uint64_t client_id);`
But in module.c, RM_DeauthenticateAndCloseClient returns int
`int RM_DeauthenticateAndCloseClient(RedisModuleCtx *ctx, uint64_t client_id)`

It it safe to change return value from `void` to `int` from the user's perspective.

(cherry picked from commit d6220f12a9)
2020-09-01 09:27:58 +03:00
Wang Yuan 9fd0c027e7 Print error info if failed opening config file (#6943)
(cherry picked from commit ea7eeb2fd2)
2020-09-01 09:27:58 +03:00
Wen Hui 0e581e0044 fix memory leak in ACLLoadFromFile error handling (#7623)
(cherry picked from commit 3f67b03378)
2020-09-01 09:27:58 +03:00
Oran Agra e8801f7eaa redis-cli --cluster-yes - negate force flag for clarity
this internal flag is there so that some commands do not comply to `--cluster-yes`

(cherry picked from commit 1aa31e4da9)
2020-09-01 09:27:58 +03:00
Frank Meier 3a677b45ac reintroduce REDISCLI_CLUSTER_YES env variable in redis-cli
the variable was introduced only in the 5.0 branch in #5879 bc6c1c40db

(cherry picked from commit 51077c8212)
2020-09-01 09:27:58 +03:00
Frank Meier dc46f6f041 add force option to 'create-cluster create' script call (#7612)
(cherry picked from commit 3244fae7d4)
2020-09-01 09:27:58 +03:00
Oran Agra 916b215fc5 fix new rdb test failing on timing issues (#7604)
apparenlty on github actions sometimes 500ms is not enough

(cherry picked from commit 824bd2ac11)
2020-09-01 09:27:58 +03:00
Yossi Gottlieb 9f0e8f7849 Fix test-centos7-tls daily job. (#7598)
(cherry picked from commit f2da3efc78)
2020-09-01 09:27:58 +03:00
Oran Agra a5294c4e52 module hook for master link up missing on successful psync (#7584)
besides, hooks test was time sensitive. when the replica managed to
reconnect quickly after the client kill, the test would fail

(cherry picked from commit f7e7775990)
2020-09-01 09:27:58 +03:00
Yossi Gottlieb e511ab2cc6 CI: Add daily CentOS 7.x jobs. (#7582)
(cherry picked from commit 4ac1f9ac55)
2020-09-01 09:27:58 +03:00
WuYunlong 7200b3aa0f Fix running single test 14-consistency-check.tcl (#7587)
(cherry picked from commit f3352daf4f)
2020-09-01 09:27:58 +03:00
Yossi Gottlieb e8efcdee0a Clarify RM_BlockClient() error condition. (#6093)
(cherry picked from commit 7af05f07ff)
2020-09-01 09:27:58 +03:00
namtsui 2bcc056a41 Avoid an out-of-bounds read in the redis-sentinel (#7443)
The Redis sentinel would crash with a segfault after a few minutes
because it tried to read from a page without read permissions. Check up
front whether the sds is long enough to contain redis:slave or
redis:master before memcmp() as is done everywhere else in
sentinelRefreshInstanceInfo().

Bug report and commit message from Theo Buehler. Fix from Nam Nguyen.

Co-authored-by: Nam Nguyen <namn@berkeley.edu>
(cherry picked from commit 63dae52324)
2020-09-01 09:27:58 +03:00
Wen Hui b2bd3c0653 Add SignalModifiedKey hook in XGROUP CREATE with MKSTREAM option (#7562)
(cherry picked from commit f33acb3f02)
2020-09-01 09:27:58 +03:00
Wen Hui 4238df5436 fix leak in error handling of debug populate command (#7062)
valsize was not modified during the for loop below instead of getting from c->argv[4], therefore there is no need to put inside the for loop.. Moreover, putting the check outside loop will also avoid memory leaking, decrRefCount(key) should be called in the original code if we put the check in for loop

(cherry picked from commit c69a9b2f61)
2020-09-01 09:27:58 +03:00
Yossi Gottlieb c68b3908a6 Fix TLS cluster tests. (#7578)
Fix consistency test added in af5167b7f without considering TLS
redis-cli configuration.

(cherry picked from commit bedf1b2126)
2020-09-01 09:27:58 +03:00
Yossi Gottlieb 1c05b87e30 TLS: Propagate and handle SSL_new() failures. (#7576)
The connection API may create an accepted connection object in an error
state, and callers are expected to check it before attempting to use it.

Co-authored-by: mrpre <mrpre@163.com>
(cherry picked from commit 784ceeb90d)
2020-09-01 09:27:58 +03:00
Oran Agra 67750ce3b3 Fix failing tests due to issues with wait_for_log_message (#7572)
- the test now waits for specific set of log messages rather than wait for
  timeout looking for just one message.
- we don't wanna sample the current length of the log after an action, due
  to a race, we need to start the search from the line number of the last
  message we where waiting for.
- when attempting to trigger a full sync, use multi-exec to avoid a race
  where the replica manages to re-connect before we completed the set of
  actions that should force a full sync.
- fix verify_log_message which was broken and unused

(cherry picked from commit 109b5ccdcd)
2020-09-01 09:27:58 +03:00
Jiayuan Chen 096285ab64 Add optional tls verification (#7502)
Adds an `optional` value to the previously boolean `tls-auth-clients` configuration keyword.

Co-authored-by: Yossi Gottlieb <yossigo@gmail.com>
(cherry picked from commit f31260b044)
2020-09-01 09:27:58 +03:00
Oran Agra 0b2f7c6407 Daily github action: run cluster and sentinel tests with tls (#7575)
(cherry picked from commit 6d92eee69b)
2020-09-01 09:27:58 +03:00
Yossi Gottlieb 8a4240b1fa TLS: support cluster/replication without tls-port.
Initialize and configure OpenSSL even when tls-port is not used, because
we may still have tls-cluster or tls-replication.

Also, make sure to reconfigure OpenSSL when these parameters are changed
as TLS could have been enabled for the first time.

(cherry picked from commit c75512d89d)
2020-09-01 09:27:58 +03:00
grishaf 395d339234 Fix prepareForShutdown function declaration (#7566)
(cherry picked from commit 4126ca466f)
2020-09-01 09:27:58 +03:00
Oran Agra 6daa8b9adb Stabilize bgsave test that sometimes fails with valgrind (#7559)
on ci.redis.io the test fails a lot, reporting that bgsave didn't end.
increaseing the timeout we wait for that bgsave to get aborted.
in addition to that, i also verify that it indeed got aborted by
checking that the save counter wasn't reset.

add another test to verify that a successful bgsave indeed resets the
change counter.

(cherry picked from commit 8a57969fd7)
2020-09-01 09:27:58 +03:00
Madelyn Olson e00b6248c9 Properly reset errno for rdbLoad (#7542)
(cherry picked from commit 818dc3a089)
2020-09-01 09:27:58 +03:00
Oran Agra e8aa5583d0 testsuite may leave servers alive on error (#7549)
in cases where you have
test name {
  start_server {
    start_server {
      assert
    }
  }
}

the exception will be thrown to the test proc, and the servers are
supposed to be killed on the way out. but it seems there was always a
bug of not cleaning the server stack, and recently (#7404) we started
relying on that stack in order to kill them, so with that bug sometimes
we would have tried to kill the same server twice, and leave one alive.

luckly, in most cases the pattern is:
start_server {
  test name {
  }
}

(cherry picked from commit 36b9494385)
2020-09-01 09:27:58 +03:00
Yossi Gottlieb f1d5d5d28e Tests: drop TCL 8.6 dependency. (#7548)
This re-implements the redis-cli --pipe test so it no longer depends on a close feature available only in TCL 8.6.

Basically what this test does is run redis-cli --pipe, generates a bunch of commands and pipes them through redis-cli, and inspects the result in both Redis and the redis-cli output.

To do that, we need to close stdin for redis-cli to indicate we're done so it can flush its buffers and exit. TCL has bi-directional channels can only offers a way to "one-way close" a channel with TCL 8.6. To work around that, we now generate the commands into a file and feed that file to redis-cli directly.

As we're writing to an actual file, the number of commands is now reduced.

(cherry picked from commit f57e844b2e)
2020-09-01 09:27:58 +03:00
Oran Agra cc7f4ba67e Fixes to release scripts (#7547)
(cherry picked from commit 343dd9bcce)
2020-09-01 09:27:58 +03:00
Remi Collet af907e4b6d Fix deprecated tail syntax in tests (#7543)
(cherry picked from commit 3f2fbc4c61)
2020-09-01 09:27:58 +03:00
Wen Hui d2f09a054c Add missing calls to raxStop (#7532)
Since the dynamic allocations in raxIterator are only used for deep walks, memory
leak due to missing call to raxStop can only happen for rax with key names longer
than 32 bytes.

Out of all the missing calls, the only ones that may lead to a leak are the rax
for consumer groups and consumers, and these were only in AOFRW and rdbSave, which
normally only happen in fork or at shutdown.

(cherry picked from commit 4e8f2d6881)
2020-09-01 09:27:58 +03:00
Wen Hui cef46edebc add missing caching command in client help (#7399)
(cherry picked from commit 2fbd0271f6)
2020-09-01 09:27:58 +03:00
zhaozhao.zz 9e997cd0d1 replication: need handle -NOPERM error after send ping (#7538)
(cherry picked from commit 13e50935a8)
2020-09-01 09:27:58 +03:00
Scott Brenner 171aa22b0b GitHub Actions workflows - use latest version of actions/checkout (#7534)
(cherry picked from commit 2f4e9c3f9f)
2020-09-01 09:27:58 +03:00
John Sully f17d178d03 Add build flag to disable MVCC tstamps 2020-08-10 06:10:24 +00:00
John Sully d88fd1588d Fix lock after free in module API 2020-08-10 06:10:24 +00:00
Oran Agra 7bf665f125 Redis 6.0.6. 2020-07-20 21:08:26 +03:00
Oran Agra a5696bdf4f Run daily CI on PRs to release a branch 2020-07-20 21:08:26 +03:00
Itamar Haber e15528bf1d Adds SHA256SUM to redis-stable tarball upload
(cherry picked from commit 5df0a64d30)
2020-07-20 21:08:26 +03:00
yoav-steinberg e28aa99af1 Support passing stack allocated module strings to moduleCreateArgvFromUserFormat (#7528)
Specifically, the key passed to the module aof_rewrite callback is a stack allocated robj. When passing it to RedisModule_EmitAOF (with appropriate "s" fmt string) redis used to panic when trying to inc the ref count of the stack allocated robj. Now support such robjs by coying them to a new heap robj. This doesn't affect performance because using the alternative "c" or "b" format strings also copies the input to a new heap robj.

(cherry picked from commit d484b8a04e)
2020-07-20 21:08:26 +03:00
Luke Palmer 3051430040 Send null for invalidate on flush (#7469)
(cherry picked from commit 5f716ea467)
2020-07-20 21:08:26 +03:00
dmurnane 29b20fd528 Notify systemd on sentinel startup (#7168)
Co-authored-by: Daniel Murnane <dmurnane@eitccorp.com>
(cherry picked from commit 9242ccf238)
2020-07-20 21:08:26 +03:00
Developer-Ecosystem-Engineering 5b36681213 Add registers dump support for Apple silicon (#7453)
Export following environment variables before building on macOS on Apple silicon

export ARCH_FLAGS="-arch arm64"
export SDK_NAME=macosx
export SDK_PATH=$(xcrun --show-sdk-path --sdk $SDK_NAME)
export CFLAGS="$ARCH_FLAGS -isysroot $SDK_PATH -I$SDK_PATH/usr/include"
export CXXFLAGS=$CFLAGS
export LDFLAGS="$ARCH_FLAGS"
export CC="$(xcrun -sdk $SDK_PATH --find clang) $CFLAGS"
export CXX="$(xcrun -sdk $SDK_PATH --find clang++) $CXXFLAGS"
export LD="$(xcrun -sdk $SDK_PATH --find ld) $LDFLAGS"

make
make test
..
All tests passed without errors!

Backtrack logging assumes x86 and required updating

(cherry picked from commit c2b5f1c15b)
2020-07-20 21:08:26 +03:00
Wen Hui 0f75036c07 correct error msg for num connections reaching maxclients in cluster mode (#7444)
(cherry picked from commit d85af4d6f5)
2020-07-20 21:08:26 +03:00
WuYunlong b1a01fda91 Fix command help for unexpected options (#7476)
(cherry picked from commit 93bdbf5aa4)
2020-07-20 21:08:26 +03:00
WuYunlong 83f55f61a6 Refactor RM_KeyType() by using macro. (#7486)
(cherry picked from commit dc690161d5)
2020-07-20 21:08:26 +03:00
Oran Agra f89f50dbd0 diskless master disconnect replicas when rdb child failed (#7518)
in case the rdb child failed, crashed or terminated unexpectedly redis
would have marked the replica clients with repl_put_online_on_ack and
then kill them only after a minute when no ack was received.

it would not stream anything to these connections, so the only effect of
this bug is a delay of 1 minute in the replicas attempt to re-connect.

(cherry picked from commit a176cb56a3)
2020-07-20 21:08:26 +03:00
Oran Agra 05f8975d21 redis-cli tests, fix valgrind timing issue (#7519)
this test when run with valgrind on github actions takes 160 seconds

(cherry picked from commit 254c962554)
2020-07-20 21:08:26 +03:00
WuYunlong 4780cc5efa Fix out of update help info in tcl tests. (#7516)
Before this commit, the output of "./runtest-cluster --help" is incorrect.
After this commit, the format of the following 3 output is consistent:
./runtest --help
./runtest-cluster --help
./runtest-sentinel --help

(cherry picked from commit 8128d39737)
2020-07-20 21:08:26 +03:00
Qu Chen 9bc6af2449 Replica always reports master's config epoch in CLUSTER NODES output. (#7235)
(cherry picked from commit 938c35302f)
2020-07-20 21:08:26 +03:00
Oran Agra c4b428a388 RESTORE ABSTTL skip expired keys - leak (#7511)
(cherry picked from commit 6a81450144)
2020-07-20 21:08:26 +03:00
Oran Agra aea4db2f5a fix recently added time sensitive tests failing with valgrind (#7512)
interestingly the latency monitor test fails because valgrind is slow
enough so that the time inside PEXPIREAT command from the moment of
the first mstime() call to get the basetime until checkAlreadyExpired
calls mstime() again is more than 1ms, and that test was too sensitive.

using this opportunity to speed up the test (unrelated to the failure)
the fix is just the longer time passed to PEXPIRE.

(cherry picked from commit e5227aab89)
2020-07-20 21:08:26 +03:00
Oran Agra b5c5f870a4 runtest --stop pause stops before terminating the redis server (#7513)
in the majority of the cases (on this rarely used feature) we want to
stop and be able to connect to the shard with redis-cli.
since these are two different processes interracting with the tty we
need to stop both, and we'll have to hit enter twice, but it's not that
bad considering it is rarely used.

(cherry picked from commit 02ef355f98)
2020-07-20 21:08:26 +03:00
Oran Agra 23bf3d1cb9 update release scripts for new hosts, and CI to run more tests (#7480)
* update daily CI to include cluster and sentinel tests
* update daily CI to run when creating a new release
* update release scripts to work on the new redis.io hosts

(cherry picked from commit 7f19a04f0f)
2020-07-20 21:08:26 +03:00
jimgreen2013 1be6dbbdf6 fix description about ziplist, the code is ok (#6318)
* fix description about ZIP_BIG_PREVLEN(the code is ok), it's similar to
antirez#4705

* fix description about ziplist entry encoding field (the code is ok),
the max length should be 2^32 - 1 when encoding is 5 bytes

(cherry picked from commit 67660881ed)
2020-07-20 21:08:26 +03:00
马永泽 1acdc84d10 fix benchmark in cluster mode fails to authenticate (#7488)
Co-authored-by: Oran Agra <oran@redislabs.com> (styling)
(cherry picked from commit 279b4a1464)
2020-07-20 21:08:26 +03:00
Abhishek Soni 52929b61f8 fix: typo in CI job name (#7466)
(cherry picked from commit d5648d617e)
2020-07-20 21:08:26 +03:00
Jiayuan Chen b0faf6113e Fix typo in deps README (#7500)
(cherry picked from commit af39d750cd)
2020-07-20 21:08:26 +03:00
WuYunlong f838df92d2 Add missing latency-monitor tcl test to test_helper.tcl. (#6782)
(cherry picked from commit d792db7948)
2020-07-20 21:08:26 +03:00
Yossi Gottlieb 7a536c2912 TLS: Session caching configuration support. (#7420)
* TLS: Session caching configuration support.
* TLS: Remove redundant config initialization.

(cherry picked from commit 3e6f2b1a45)
2020-07-20 21:08:26 +03:00
Yossi Gottlieb 6f1e828454 TLS: Ignore client cert when tls-auth-clients off. (#7457)
(cherry picked from commit 5266293a0f)
2020-07-20 21:08:26 +03:00
James Hilliard 2d3687655b Use pkg-config to properly detect libssl and libcrypto libraries (#7452)
(cherry picked from commit 6a014af79a)
2020-07-20 21:08:26 +03:00
Yossi Gottlieb b057ff81ee TLS: Add missing redis-cli options. (#7456)
* Tests: fix and reintroduce redis-cli tests.

These tests have been broken and disabled for 10 years now!

* TLS: add remaining redis-cli support.

This adds support for the redis-cli --pipe, --rdb and --replica options
previously unsupported in --tls mode.

* Fix writeConn().

(cherry picked from commit d9f970d8d3)
2020-07-20 21:08:26 +03:00
Oran Agra a9d1014fb2 redis-cli --hotkeys fixed to handle non-printable key names
(cherry picked from commit b23e251036)
2020-07-20 21:08:26 +03:00
Oran Agra 183bdc11c0 redis-cli --bigkeys fixed to handle non-printable key names
(cherry picked from commit 6f8a8647de)
2020-07-20 21:08:26 +03:00
Oran Agra 95ba01b538 RESTORE ABSTTL won't store expired keys into the db (#7472)
Similarly to EXPIREAT with TTL in the past, which implicitly deletes the
key and return success, RESTORE should not store key that are already
expired into the db.
When used together with REPLACE it should emit a DEL to keyspace
notification and replication stream.

(cherry picked from commit 5977a94842)
2020-07-20 21:08:26 +03:00
huangzhw 7b21b8c3fb defrag.c activeDefragSdsListAndDict when defrag sdsele, We can't use (#7492)
it to calculate hash, we should use newsds.

(cherry picked from commit d6180c8c86)
2020-07-20 21:08:26 +03:00
Oran Agra 33ca884cc5 skip a test that uses +inf on valgrind (#7440)
On some platforms strtold("+inf") with valgrind returns a non-inf result

[err]: INCRBYFLOAT does not allow NaN or Infinity in tests/unit/type/incr.tcl
Expected 'ERR*would produce*' to equal or match '1189731495357231765085759.....'

(cherry picked from commit 909bc97c52)
2020-07-20 21:08:26 +03:00
Oran Agra 2b5f23197c stabilize tests that look for log lines (#7367)
tests were sensitive to additional log lines appearing in the log
causing the search to come empty handed.

instead of just looking for the n last log lines, capture the log lines
before performing the action, and then search from that offset.

(cherry picked from commit 8e76e13472)
2020-07-20 21:08:26 +03:00
Oran Agra 1104113c07 tests/valgrind: don't use debug restart (#7404)
* tests/valgrind: don't use debug restart

DEBUG REATART causes two issues:
1. it uses execve which replaces the original process and valgrind doesn't
   have a chance to check for errors, so leaks go unreported.
2. valgrind report invalid calls to close() which we're unable to resolve.

So now the tests use restart_server mechanism in the tests, that terminates
the old server and starts a new one, new PID, but same stdout, stderr.

since the stderr can contain two or more valgrind report, it is not enough
to just check for the absence of leaks, we also need to check for some known
errors, we do both, and fail if we either find an error, or can't find a
report saying there are no leaks.

other changes:
- when killing a server that was already terminated we check for leaks too.
- adding DEBUG LEAK which was used to test it.
- adding --trace-children to valgrind, although no longer needed.
- since the stdout contains two or more runs, we need slightly different way
  of checking if the new process is up (explicitly looking for the new PID)
- move the code that handles --wait-server to happen earlier (before
  watching the startup message in the log), and serve the restarted server too.

* squashme - CR fixes

(cherry picked from commit 69ade87325)
2020-07-20 21:08:26 +03:00
Oran Agra 62f8583445 change references to the github repo location (#7479)
(cherry picked from commit 9bbf768d3c)
2020-07-20 21:08:26 +03:00
zhaozhao.zz 3155adb299 BITOP: propagate only when it really SET or DEL targetkey (#5783)
For example:
    BITOP not targetkey sourcekey

If targetkey and sourcekey doesn't exist, BITOP has no effect,
we do not propagate it, thus can save aof and replica flow.

(cherry picked from commit 1978f996d8)
2020-07-20 21:08:26 +03:00
antirez 14a59d4ce7 Update comment to clarify change in #7398.
(cherry picked from commit ad0a9df77a)
2020-07-20 21:08:26 +03:00
antirez 17aaf5ec97 LPOS: option FIRST renamed RANK.
(cherry picked from commit a5a3a7bbc6)
2020-07-20 21:08:26 +03:00
Dave Nielsen 3f5cf24da9 updated copyright year
Changed "2015" to "2020"

(cherry picked from commit 2d6d9f7552)
2020-07-20 21:08:26 +03:00
Oran Agra 05e483cbb3 EXEC always fails with EXECABORT and multi-state is cleared
In order to support the use of multi-exec in pipeline, it is important that
MULTI and EXEC are never rejected and it is easy for the client to know if the
connection is still in multi state.

It was easy to make sure MULTI and DISCARD never fail (done by previous
commits) since these only change the client state and don't do any actual
change in the server, but EXEC is a different story.

Since in the past, it was possible for clients to handle some EXEC errors and
retry the EXEC, we now can't affort to return any error on EXEC other than
EXECABORT, which now carries with it the real reason for the abort too.

Other fixes in this commit:
- Some checks that where performed at the time of queuing need to be re-
  validated when EXEC runs, for instance if the transaction contains writes
  commands, it needs to be aborted. there was one check that was already done
  in execCommand (-READONLY), but other checks where missing: -OOM, -MISCONF,
  -NOREPLICAS, -MASTERDOWN
- When a command is rejected by processCommand it was rejected with addReply,
  which was not recognized as an error in case the bad command came from the
  master. this will enable to count or MONITOR these errors in the future.
- make it easier for tests to create additional (non deferred) clients.
- add tests for the fixes of this commit.

(cherry picked from commit 65a3307bc9)
2020-07-20 21:08:26 +03:00
antirez c8f250f8dd Include cluster.h for getClusterConnectionsCount().
(cherry picked from commit 21f62c3346)
2020-07-20 21:08:26 +03:00
antirez 9412094f25 Fix BITFIELD i64 type handling, see #7417.
(cherry picked from commit 746297314f)
2020-07-20 21:08:26 +03:00
antirez 8312aa27d4 Clarify maxclients and cluster in conf. Remove myself too.
(cherry picked from commit 59fd178014)
2020-07-20 21:08:26 +03:00
hwware d3aa3791fe fix memory leak in sentinel connection sharing
(cherry picked from commit 1bfa2d27a6)
2020-07-20 21:08:26 +03:00
chenhui0212 2b189f098e Fix comments in function raxLowWalk of listpack.c
(cherry picked from commit f800b52172)
2020-07-20 21:08:26 +03:00
Tomasz Poradowski 3328b7a514 ensure SHUTDOWN_NOSAVE in Sentinel mode
- enforcing of SHUTDOWN_NOSAVE flag in one place to make it consitent
  when running in Sentinel mode

(cherry picked from commit 4ee011adb5)
2020-07-20 21:08:26 +03:00
chenhui0212 88e1f80eee fix comments in listpack.c
(cherry picked from commit 71fafd761a)
2020-07-20 21:08:26 +03:00
antirez 0ebbc36059 Use cluster connections too, to limit maxclients.
See #7401.

(cherry picked from commit 4b8d8826af)
2020-07-20 21:08:26 +03:00
antirez 43ed3c3589 Tracking: fix enableBcastTrackingForPrefix() invalid sdslen() call.
Related to #7387.

(cherry picked from commit 784479939d)
2020-07-20 21:08:26 +03:00
root 8095daea4a cluster.c remove if of clusterSendFail in markNodeAsFailingIfNeeded
(cherry picked from commit c92464db69)
2020-07-20 21:08:26 +03:00
meir@redislabs.com 51e178454d Fix RM_ScanKey module api not to return int encoded strings
The scan key module API provides the scan callback with the current
field name and value (if it exists). Those arguments are RedisModuleString*
which means it supposes to point to robj which is encoded as a string.
Using createStringObjectFromLongLong function might return robj that
points to an integer and so break a module that tries for example to
use RedisModule_StringPtrLen on the given field/value.

The PR introduces a fix that uses the createObject function and sdsfromlonglong function.
Using those function promise that the field and value pass to the to the
scan callback will be Strings.

The PR also changes the Scan test module to use RedisModule_StringPtrLen
to catch the issue. without this, the issue is hidden because
RedisModule_ReplyWithString knows to handle integer encoding of the
given robj (RedisModuleString).

The PR also introduces a new test to verify the issue is solved.

(cherry picked from commit a89bf734a9)
2020-07-20 21:08:26 +03:00
antirez 82b2bfd20b Fix LCS object type checking. Related to #7379.
(cherry picked from commit 1055398849)
2020-07-20 21:08:26 +03:00
hwware ec1faeec72 fix memory leak
(cherry picked from commit 7008a0ba66)
2020-07-20 21:08:26 +03:00
hwware c1326d7b10 fix server crash in STRALGO command
(cherry picked from commit 2a05fa0d48)
2020-07-20 21:08:26 +03:00
Benjamin Sergeant 845fb2d1c1 Update redis-cli.c
(cherry picked from commit 93021da221)
2020-07-20 21:08:26 +03:00
Jamie Scott 930fac8058 minor fix
(cherry picked from commit 4f3b15e6ea)
2020-07-20 21:08:26 +03:00
John Sully 119797014c Remove gitter, we don't check it often enough 2020-07-13 21:14:03 -04:00
John Sully e8f568644b Merge branch 'unstable' of https://github.com/JohnSully/KeyDB into unstable 2020-07-13 01:05:17 +00:00
John Sully b736a95b0d Merge tag '6.0.5' into unstable
Redis 6.0.5
2020-07-13 00:55:23 +00:00
benschermel d909d12cb3 Merge branch 'unstable' of https://github.com/JohnSully/KeyDB into unstable 2020-07-12 18:04:40 -04:00
benschermel 201ffa9d62 update rpm permissions 2020-07-12 18:03:21 -04:00
John Sully 5619f51528 Add the KEYDB.MEXISTS command, see issue #203 2020-07-12 21:42:11 +00:00
John Sully d99d06b125 Add multi-master-no-forward command to reduce bus traffic with multi-master 2020-07-12 19:25:19 +00:00
John Sully 1b9ade7f65 Document min-clients-per-thread 2020-07-12 18:49:02 +00:00
John Sully e01312642b Prevent deadlock in RM_ThreadSafeContextLock() when triggered as part of a module callback in a server thread 2020-07-12 18:17:53 +00:00
John Sully cd9514f4c8 Fix failure to merge databases on active replica sync, due to bad merge with Redis 6 2020-07-12 01:13:22 +00:00
John Sully f111c23415 Support missing Redis 6 config options 2020-07-11 21:39:20 +00:00
John Sully 6857c4c208 Latency fixes 2020-07-01 21:59:38 -04:00
Muhammad Zahalqa 52897f8f9a replace memcpy with cctor on objects 2020-06-24 23:20:59 -04:00
Muhammad Zahalqa 8bc70e9840 unique lock used with cv.wait already owns mutex 2020-06-24 23:18:53 -04:00
Muhammad Zahalqa 5b81303f04 Fix bug in condition_variabe.wait usage
consition variable wait should always be called with a locked mutex.
This code break the pre-conditions of using wait.
2020-06-24 23:18:53 -04:00
Muhammad Zahalqa ede524c064 fixes for robj_sharedptr
1. fix cases where null pointer might be accessed
2. make assignmnet op safe
3. make operator bool explicit (safe bool idiom)
4. make comparison operators symetric

fix robj_sharedptr use in rdb.cpp
2020-06-24 23:18:27 -04:00
Muhammad Zahalqa aeed09c346 aeCommand objects no need for memset and missing init of some memebers. 2020-06-24 23:15:29 -04:00
antirez 0cae0900cf help.h updated. 2020-06-12 12:16:24 +02:00
antirez e0022d8cfe LPOS: tests + crash fix. 2020-06-12 12:08:06 +02:00
antirez 495327c020 LPOS: update to latest proposal.
See https://gist.github.com/antirez/3591c5096bc79cad8b5a992e08304f48
2020-06-12 12:08:06 +02:00
antirez afb8b3c030 LPOS: implement the final design. 2020-06-12 12:08:06 +02:00
Paul Spooren 31c9cd8604 LRANK: Add command (the command will be renamed LPOS).
The `LRANK` command returns the index (position) of a given element
within a list. Using the `direction` argument it is possible to specify
going from head to tail (acending, 1) or from tail to head (decending,
-1). Only the first found index is returend. The complexity is O(N).

When using lists as a queue it can be of interest at what position a
given element is, for instance to monitor a job processing through a
work queue. This came up within the Python `rq` project which is based
on Redis[0].

[0]: https://github.com/rq/rq/issues/1197

Signed-off-by: Paul Spooren <mail@aparcar.org>
2020-06-12 12:08:06 +02:00
John Sully 70b80aa5fa Implement replicaof remove as requested in issue #192 2020-06-09 12:43:58 -04:00
antirez 51efb7fe25 Redis 6.0.5. 2020-06-09 12:19:30 +02:00
antirez a7d3670da5 Adapt EVAL+busy script test to new behavior. 2020-06-09 12:19:30 +02:00
antirez ee8dd01bbc Temporary fix for #7353 issue about EVAL during -BUSY. 2020-06-09 11:53:01 +02:00
xhe a4a856d532 return the correct proto version
HELLO should return the current proto version, while the code hardcoded
3
2020-06-09 11:53:01 +02:00
Oran Agra e2046b3003 Don't queue commands in an already aborted MULTI state 2020-06-09 11:53:01 +02:00
Oran Agra b35fdf1de1 Avoid rejecting WATCH / UNWATCH, like MULTI/EXEC/DISCARD
Much like MULTI/EXEC/DISCARD, the WATCH and UNWATCH are not actually
operating on the database or server state, but instead operate on the
client state. the client may send them all in one long pipeline and check
all the responses only at the end, so failing them may lead to a
mismatch between the client state on the server and the one on the
client end, and execute the wrong commands (ones that were meant to be
discarded)

the watched keys are not actually stored in the client struct, but they
are in fact part of the client state. for instance, they're not cleared
or moved in SWAPDB or FLUSHDB.
2020-06-09 11:53:01 +02:00
zhaozhao.zz 1d7bf208ce AOF: append origin SET if no expire option 2020-06-09 11:53:01 +02:00
Oran Agra 676445ad95 fix disconnectSlaves, to try to free each slave.
the recent change in that loop (iteration rather than waiting for it to
be empty) was intended to avoid an endless loop in case some slave would
refuse to be freed.

but the lookup of the first client remained, which would have caused it
to try the first one again and again instead of moving on.
2020-06-09 11:53:01 +02:00
zhaozhao.zz 4846c0c8af donot free protected client in freeClientsInAsyncFreeQueue
related #7234
2020-06-09 11:53:01 +02:00
John Sully e8d4d095cc Update gcov for C++ 2020-06-08 16:48:44 -04:00
Ben Schermel c50a3568a5 update Dockerfile 2020-06-08 15:34:22 -04:00
Ben Schermel 17e96b91b5 updating changelog 2020-06-08 15:34:22 -04:00
Ben Schermel c5e7a8517b dh9 for stretch deb pkg 2020-06-08 15:34:22 -04:00
Ben Schermel c36773777d fix missing sentinel 2020-06-08 15:34:22 -04:00
Ben Schermel d69e0b8561 update changelog 2020-06-08 15:34:22 -04:00
Ben Schermel 19ccc22a43 update changelog 2020-06-08 15:34:22 -04:00
Ben Schermel 6968a0cfcb update changelog 2020-06-08 15:34:22 -04:00
Ben Schermel df618fbd9f include support for dists < dh11 2020-06-08 15:34:22 -04:00
Ben Schermel 94acd43340 fix rpmbuild dirs 2020-06-08 15:34:22 -04:00
Ben Schermel 14bdd745d1 fix rpmbuild dirs 2020-06-08 15:34:22 -04:00
Ben Schermel aa3f5de16f adding dh_installsystemd 2020-06-08 15:34:22 -04:00
Ben Schermel 9f7890cb1a strip unused from deb source 2020-06-08 15:34:22 -04:00
benschermel 1dc34c16b5 add deb/rpm/docker support 2020-06-08 15:34:22 -04:00
benschermel b0e7be7ce2 add deb/rpm/docker support 2020-06-08 15:34:22 -04:00
benschermel 026ab4c922 update github CI for deb 2020-06-08 15:34:22 -04:00
Ben Schermel 07ba64c200 update rpm versioning 2020-06-08 15:34:22 -04:00
benschermel b4b35cf904 add package utilities 2020-06-08 15:34:22 -04:00
John Sully ba974c9810 Some valgrind fixes 2020-06-07 23:20:10 -04:00
John Sully 9817db7edd fix valgrind false error 2020-06-07 20:17:47 -04:00
John Sully 515bb0e6bd Endurance mode for tests ignore flaky ones 2020-06-07 16:59:59 -04:00
John Sully bc7ea8fb20 Solo tests should work with loopn 2020-06-07 15:06:50 -04:00
John Sully 2caf8e4c80 Stream tests rely on deferring clients, mitigate races caused by them 2020-06-07 13:59:41 -04:00
John Sully e1f3cd6ec3 replication test race 2020-06-07 01:14:57 -04:00
John Sully 02e821530a fix race in test 2020-06-07 00:44:11 -04:00
John Sully b6c3d80edb Make lazyfree more reliable 2020-06-07 00:40:58 -04:00
Oran Agra f33de403ed fix pingoff test race 2020-06-06 11:44:21 +02:00
Kevin Fwu 49af4d07e4 Fix TLS certificate loading for chained certificates.
This impacts client verification for chained certificates (such as Lets
Encrypt certificates). Client Verify requires the full chain in order to
properly verify the certificate.
2020-06-06 11:44:21 +02:00
antirez 329fddbda1 Revert "Implements sendfile for redis."
This reverts commit 9cf500a3f6.
2020-06-06 11:43:29 +02:00
antirez 925a2cd5aa Revert "avoid using sendfile if tls-replication is enabled"
This reverts commit b9abecfc4c.
2020-06-06 11:43:29 +02:00
Liu Zhen 84a7a90586 fix clusters mixing accidentally by gossip
`clusterStartHandshake` will start hand handshake
and eventually send CLUSTER MEET message, which is strictly prohibited
in the REDIS CLUSTER SPEC.
Only system administrator can initiate CLUSTER MEET message.
Futher, according to the SPEC, rather than IP/PORT pairs, only nodeid
can be trusted.
2020-06-06 11:43:29 +02:00
antirez cd63359a17 Fix handling of special chars in ACL LOAD.
Now it is also possible for ACL SETUSER to accept empty strings
as valid operations (doing nothing), so for instance

    ACL SETUSER myuser ""

Will have just the effect of creating a user in the default state.

This should fix #7329.
2020-06-06 11:43:29 +02:00
John Sully 0042586190 Detect issues like #189 2020-06-05 22:18:28 -04:00
John Sully c6dce838b7 Ensure CRON jobs run in a clean environment 2020-06-05 21:35:47 -04:00
John Sully 5b885bb164 Undecorated new is OK to use 2020-06-01 23:36:01 -04:00
John Sully 95f46e6077 maxmem tests should be run solo for higher reliability 2020-06-01 16:50:30 -04:00
John Sully 31f58311e3 Remove debug logs that shouldn't have been checked in 2020-06-01 16:34:05 -04:00
John Sully 130613e166 PSYNC test shouldn't wait forever 2020-06-01 16:13:58 -04:00
John Sully 50fd4fa7e6 PSYNC test reliability improvements (test only issue) 2020-06-01 16:01:26 -04:00
John Sully e6a9a6b428 Fix for issue #187 we need to properly handle the case where a key with a subkey expirey itself expires during load 2020-06-01 15:33:19 -04:00
John Sully d6be93537e Update endurance.yml 2020-06-01 00:25:31 -04:00
John Sully 1ef35cf466 Fix module multithreaded test failures 2020-05-31 23:46:12 -04:00
John Sully 1eb0e3c1c6 sendfile has high latency in some scenarios, don't use it 2020-05-31 23:22:25 -04:00
John Sully e50313b207 we must always respect the output of sdscat, also change the string so its not a substring of other config params 2020-05-31 23:22:10 -04:00
John Sully ac3f8d1ac7 Do another iteration for endurance 2020-05-31 22:51:02 -04:00
John Sully 202b97eff7 Don't start multimaster tests until all nodes are connected 2020-05-31 22:50:30 -04:00
John Sully f25aebf269 Fix client race in test (test only issue) 2020-05-31 21:39:50 -04:00
John Sully 9b1e636a66 --loop should run forever 2020-05-31 21:27:15 -04:00
John Sully 5147f0153a PUBSUB test reliability: A client race was erroneously failing tests (test only issue) 2020-05-31 21:27:15 -04:00
John Sully 88f7739ae0 Update endurance.yml 2020-05-31 17:50:02 -04:00
John Sully c0c033a0c1 Active replicas referencing eachother should connect one at a time 2020-05-31 16:03:50 -04:00
John Sully 73300c5700 Fix race in futex_sleep 2020-05-31 16:03:50 -04:00
John Sully 37033179fe Update endurance.yml 2020-05-31 14:18:12 -04:00
John Sully 3e6a5ac9cf Update endurance.yml 2020-05-31 03:43:06 -04:00
John Sully dcd297eb5b Endurance tests need more cores 2020-05-30 22:16:56 -04:00
John Sully 4a9742e696 Update endurance.yml 2020-05-30 20:03:04 -04:00
John Sully 8edbae2e04 Auto tune lock for high CPU tension scenarios 2020-05-30 18:22:27 -04:00
John Sully 63c6735ffc Ensure endurance run can complete in an hour 2020-05-30 15:31:54 -04:00
John Sully 8a0441c144 We only want one master connection in progress at a time, ensure that if the connection fails we try a different master 2020-05-30 15:13:28 -04:00
John Sully 460c5ed2b3 daily CI config fix 2020-05-30 03:27:37 -04:00
John Sully 8f44f0937b More clients for endurance 2020-05-30 03:13:04 -04:00
John Sully 1803b1587b Endurance tests need to be faster, reduce loops 2020-05-30 02:40:44 -04:00
John Sully a5325061f2 Only CI ARM daily 2020-05-30 02:34:37 -04:00
John Sully a10a0df09f Also test with multiple threads 2020-05-30 02:10:00 -04:00
John Sully 031044c7ea Reduce clients for endurance test 2020-05-30 02:07:30 -04:00
John Sully ca755e4905 Fix cron syntax 2020-05-30 01:53:21 -04:00
John Sully 3063a028b3 Add endurance CI tests 2020-05-30 01:47:26 -04:00
John Sully 2c659acc7d Update daily CI job 2020-05-30 01:21:50 -04:00
John Sully 341a8caef9 TLS requires we explicitly marshal connected sockets when using them on a different thread 2020-05-30 01:14:53 -04:00
John Sully 366f8c2dc6 Merge branch 'unstable' of https://github.com/JohnSully/KeyDB into unstable 2020-05-30 00:06:23 -04:00
John Sully 5803540422 Fix memory test failures with multithreading enabled 2020-05-30 00:06:15 -04:00
Ben Schermel f17c492a63 centos build fix 2020-05-29 22:23:48 -04:00
John Sully aad6a7ce15 sendFile blocks too long for use with active replication 2020-05-29 20:27:45 -04:00
John Sully 1cb48c7bf6 Unify beforeSleep handling 2020-05-29 17:44:55 -04:00
John Sully f25c405ad2 Cluster crash 2020-05-29 03:40:52 -04:00
John Sully c9920849dd active replica tests on slow computers 2020-05-29 01:58:15 -04:00
John Sully 81b240f81d Fix test issue with TLS 2020-05-29 01:44:52 -04:00
John Sully 6a902b29e1 Fix CI 2020-05-29 01:01:41 -04:00
John Sully 9c31ac7925 Merge tag '6.0.4' into unstable
Redis 6.0.4.
2020-05-29 00:57:07 -04:00
John Sully f0d8d1680f Fix TLS tests 2020-05-29 00:16:06 -04:00
John Sully 6dcbcacb10 gencert its own step 2020-05-28 23:49:13 -04:00
John Sully 3196717cee CI yml syntax error 2020-05-28 23:18:02 -04:00
John Sully f95578ac02 Fix TLS tests in CI 2020-05-28 23:16:42 -04:00
John Sully 57ca6facc3 fix TLS test failure 2020-05-28 23:16:27 -04:00
antirez 886d287c40 Redis 6.0.4. 2020-05-28 12:18:38 +02:00
antirez 59cd4c9f65 Test: take PSYNC2 test master timeout high during switch.
This will likely avoid false positives due to trailing pings.
2020-05-28 10:56:14 +02:00
antirez 6c1bb7b194 Test: add the tracking unit as default. 2020-05-28 10:22:29 +02:00
Oran Agra 1aee695e52 tests: find_available_port start search from next port
i.e. don't start the search from scratch hitting the used ones again.
this will also reduce the likelihood of collisions (if there are any
left) by increasing the time until we re-use a port we did use in the
past.
2020-05-28 10:09:51 +02:00
Oran Agra a2ae463520 tests: each test client work on a distinct port range
apparently when running tests in parallel (the default of --clients 16),
there's a chance for two tests to use the same port.
specifically, one test might shutdown a master and still have the
replica up, and then another test will re-use the port number of master
for another master, and then that replica will connect to the master of
the other test.

this can cause a master to count too many full syncs and fail a test if
we run the tests with --single integration/psync2 --loop --stop

see Probmem 2 in #7314
2020-05-28 10:09:51 +02:00
Oran Agra 86e562d691 32bit CI needs to build modules correctly 2020-05-28 10:09:51 +02:00
Oran Agra ab2984b1e2 adjust revived meaningful offset tests
these tests create several edge cases that are otherwise uncovered (at
least not consistently) by the test suite, so although they're no longer
testing what they were meant to test, it's still a good idea to keep
them in hope that they'll expose some issue in the future.
2020-05-28 10:09:51 +02:00
Oran Agra 1ff5a222de revive meaningful offset tests 2020-05-28 10:09:51 +02:00
antirez cc549b46a4 Replication: showLatestBacklog() refactored out. 2020-05-28 10:09:51 +02:00
antirez 377dd0515c Drop useless line from replicationCacheMaster(). 2020-05-28 10:09:51 +02:00
antirez 3f8d113f1b Another meaningful offset test removed. 2020-05-28 10:09:51 +02:00
antirez d4541349dc Remove the PSYNC2 meaningful offset test. 2020-05-28 10:09:51 +02:00
antirez 2112a57028 Remove the meaningful offset feature.
After a closer look, the Redis core devleopers all believe that this was
too fragile, caused many bugs that we didn't expect and that were very
hard to track. Better to find an alternative solution that is simpler.
2020-05-28 10:09:51 +02:00
antirez d2eb6e0b47 Set a protocol error if master use the inline protocol.
We want to react a bit more aggressively if we sense that the master is
sending us some corrupted stream. By setting the protocol error we both
ensure that the replica will disconnect, and avoid caching the master so
that a full SYNC will be required. This is protective against
replication bugs.
2020-05-28 10:09:51 +02:00
Oran Agra 9c1df3b769 daily CI test with tls 2020-05-28 10:09:51 +02:00
Oran Agra 115ed1911e avoid using sendfile if tls-replication is enabled
this obviously broke the tests, but went unnoticed so far since tls
wasn't often tested.
2020-05-28 10:09:51 +02:00
antirez 11c748aaca Replication: log backlog creation event. 2020-05-28 10:09:51 +02:00
antirez 8f10137227 Test: PSYNC2 test can now show server logs. 2020-05-28 10:09:51 +02:00
John Sully 561a1f8d8a Test TLS as part of CI 2020-05-26 01:39:54 -04:00
John Sully 1449c8b9fe Merge branch 'unstable' of https://github.com/JohnSully/KeyDB into unstable 2020-05-26 01:29:08 -04:00
John Sully 1938af27f5 Fix client tracking, also ensure tracking tests are enabled 2020-05-26 01:28:52 -04:00
antirez 2e591fc4a4 Clarify what is happening in PR #7320. 2020-05-25 12:08:01 +02:00
zhaozhao.zz cbb51fb8f9 PSYNC2: second_replid_offset should be real meaningful offset
After adjustMeaningfulReplOffset(), all the other related variable
should be updated, including server.second_replid_offset.

Or the old version redis like 5.0 may receive wrong data from
replication stream, cause redis 5.0 can sync with redis 6.0,
but doesn't know meaningful offset.
2020-05-25 12:08:01 +02:00
Oran Agra e0fc88b4d9 add CI for 32bit build 2020-05-25 12:08:01 +02:00
antirez e3f864b5fd Make disconnectSlaves() synchronous in the base case.
Otherwise we run into that:

Backtrace:
src/redis-server 127.0.0.1:21322(logStackTrace+0x45)[0x479035]
src/redis-server 127.0.0.1:21322(sigsegvHandler+0xb9)[0x4797f9]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x11390)[0x7fd373c5e390]
src/redis-server 127.0.0.1:21322(_serverAssert+0x6a)[0x47660a]
src/redis-server 127.0.0.1:21322(freeReplicationBacklog+0x42)[0x451282]
src/redis-server 127.0.0.1:21322[0x4552d4]
src/redis-server 127.0.0.1:21322[0x4c5593]
src/redis-server 127.0.0.1:21322(aeProcessEvents+0x2e6)[0x42e786]
src/redis-server 127.0.0.1:21322(aeMain+0x1d)[0x42eb0d]
src/redis-server 127.0.0.1:21322(main+0x4c5)[0x42b145]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf0)[0x7fd3738a3830]
src/redis-server 127.0.0.1:21322(_start+0x29)[0x42b409]

Since we disconnect all the replicas and free the replication backlog in
certain replication paths, and the code that will free the replication
backlog expects that no replica is connected.

However we still need to free the replicas asynchronously in certain
cases, as documented in the top comment of disconnectSlaves().
2020-05-25 12:08:01 +02:00
ShooterIT 8af1e513f6 Implements sendfile for redis. 2020-05-25 12:08:01 +02:00
antirez 3c21418cd8 Fix #7306 less aggressively.
Citing from the issue:

btw I suggest we change this fix to something else:
* We revert the fix.
* We add a call that disconnects chained replicas in the place where we trim the replica (that is a master i this case) offset.
This way we can avoid disconnections when there is no trimming of the backlog.

Note that we now want to disconnect replicas asynchronously in
disconnectSlaves(), because it's in general safer now that we can call
it from freeClient(). Otherwise for instance the command:

    CLIENT KILL TYPE master

May crash: clientCommand() starts running the linked of of clients,
looking for clients to kill. However it finds the master, kills it
calling freeClient(), but this in turn calls replicationCacheMaster()
that may also call disconnectSlaves() now. So the linked list iterator
of the clientCommand() will no longer be valid.
2020-05-25 12:08:01 +02:00
Maxime de Roucy 123bc125cf acl: @replication command group
containing mandatory commands for replication (used by masteruser user)
2020-05-25 02:35:22 -04:00
John Sully ecbe168b54 Fix issue #170. Intermittent crash destroying fastlock due to implicit type conversion in assert 2020-05-25 02:13:57 -04:00
John Sully 00cef78858 Fix infinite logs of Error in rreplay 2020-05-24 20:05:22 -04:00
John Sully d05da0fabd Enable TLS connections 2020-05-24 19:57:16 -04:00
John Sully 4db3f2a443 don't keep feeding data to clients pending close 2020-05-24 12:46:11 -04:00
John Sully fd710d590a Quiet TSAN during exit 2020-05-24 12:45:10 -04:00
John Sully 9b98c9edd3 Install deps for daily workflow 2020-05-24 11:19:53 -04:00
John Sully 4893eedacb Hold the AE lock until we process the repl backlog after a command 2020-05-24 11:19:23 -04:00
John Sully fb50163b47 Add standardized way to post client dependent functions 2020-05-24 02:46:39 -04:00
John Sully 3b8463f9d2 remove unnecessary cplusplus guard 2020-05-24 02:45:55 -04:00
John Sully 42c7677690 race in connection state when setting a handler on a different thread 2020-05-24 02:45:11 -04:00
John Sully 8665f25031 Fix ASAN races 2020-05-24 01:58:33 -04:00
John Sully 1c6b573147 Incorrect locking in replication error paths 2020-05-24 01:57:49 -04:00
John Sully 90162da1a6 Incorrect memory order on weak ordered architectures 2020-05-24 01:57:18 -04:00
John Sully 1ca6c1dd56 If we request shutdown from a client on a different thread than a blocking lua script we fail to actually shutdown until the script completes 2020-05-24 01:56:23 -04:00
John Sully 2d27c8ecdb race calculating client overhead 2020-05-24 01:53:43 -04:00
John Sully 59cda9cfb9 During RDB load we can only handle clients on the main thread as our workers are not yet created. This change also unifies the accept logic across the socket types 2020-05-23 16:03:24 -04:00
John Sully 091af64003 Diskless replication handler needs to run on the main thread 2020-05-23 15:15:42 -04:00
ShooterIT a12210c875 Fix reply bytes calculation error
Fix #7275.
2020-05-23 13:02:55 -04:00
John Sully 43649fc5b4 Merge tag '6.0.3' into unstable
Redis 6.0.3
2020-05-22 16:53:10 -04:00
John Sully 01cc25653a Merge branch 'unstable' of https://github.com/JohnSully/KeyDB into unstable 2020-05-22 16:45:27 -04:00
John Sully a010e4a4b2 Merge tag '6.0.2' into unstable
Redis 6.0.2
2020-05-22 16:45:18 -04:00
John Sully 817a11ff11 Merge commit 'cb683a84f7a9d530bec629f2b656e7d0842a0f75' into unstable 2020-05-22 15:56:35 -04:00
John Sully c52e1fc78c Merge commit '3a441c7d95c8757d2ab12ff730cc7ca8899538c9' into unstable 2020-05-22 15:42:50 -04:00
John Sully d74871da40 Merge commit 'c57d9146f41f4b661d9d2cb48b83b3abc757ba0e' into unstable 2020-05-22 15:36:44 -04:00
John Sully a25cf15ecd Update README.md 2020-05-22 14:03:29 -04:00
Madelyn Olson e201f83cee EAGAIN for tls during diskless load 2020-05-22 12:37:59 +02:00
Qu Chen 58fc456cbd Disconnect chained replicas when the replica performs PSYNC with the master always to avoid replication offset mismatch between master and chained replicas. 2020-05-22 12:37:59 +02:00
hwware 3febc5c29e using moreargs variable 2020-05-22 12:37:59 +02:00
hwware 8d67385593 fix server crash for STRALGO command 2020-05-22 12:37:49 +02:00
ShooterIT 7a35eec540 Replace addDeferredMultiBulkLength with addReplyDeferredLen in comment 2020-05-22 12:37:49 +02:00
Yossi Gottlieb f93e1417b9 TLS: Improve tls-protocols clarity in redis.conf. 2020-05-22 12:37:49 +02:00
ShooterIT d0c9e44545 Fix reply bytes calculation error
Fix #7275.
2020-05-22 12:37:49 +02:00
zhaozhao.zz 1cde6a0603 Tracking: flag CLIENT_TRACKING_BROKEN_REDIR when redir broken 2020-05-22 12:37:49 +02:00
Oran Agra 436be34986 fix a rare active defrag edge case bug leading to stagnation
There's a rare case which leads to stagnation in the defragger, causing
it to keep scanning the keyspace and do nothing (not moving any
allocation), this happens when all the allocator slabs of a certain bin
have the same % utilization, but the slab from which new allocations are
made have a lower utilization.

this commit fixes it by removing the current slab from the overall
average utilization of the bin, and also eliminate any precision loss in
the utilization calculation and move the decision about the defrag to
reside inside jemalloc.

and also add a test that consistently reproduce this issue.
2020-05-22 12:37:49 +02:00
Oran Agra f9d2ffdc5d improve DEBUG MALLCTL to be able to write to write only fields.
also support:
  debug mallctl-str thread.tcache.flush VOID
2020-05-22 12:37:49 +02:00
hujie d7968ee920 fix clear USER_FLAG_ALLCOMMANDS flag in acl
in ACLSetUserCommandBit, when the command bit overflows, no operation
is performed, so no need clear the USER_FLAG_ALLCOMMANDS flag.

in ACLSetUser, when adding subcommand, we don't need to call
ACLGetCommandID ahead since subcommand may be empty.
2020-05-22 12:37:49 +02:00
ShooterIT a902e6b251 Redis Benchmark: generate random test data
The function of generating random data is designed by antirez. See #7196.
2020-05-22 12:37:49 +02:00
hwware 9564ed7c39 Redis-Benchmark: avoid potentical memmory leaking 2020-05-22 12:37:49 +02:00
WuYunlong 2e41827435 Handle keys with hash tag when computing hash slot using tcl cluster client. 2020-05-22 12:37:49 +02:00
WuYunlong eb2c8b2c61 Add a test to prove current tcl cluster client can not handle keys with hash tag. 2020-05-22 12:37:49 +02:00
ShooterIT 928e6976bb Use dictSize to get the size of dict in dict.c 2020-05-22 12:37:49 +02:00
Madelyn Olson cdcf5af5aa Converge hash validation for adding and removing 2020-05-22 12:37:49 +02:00
Benjamin Sergeant e8b09d2203 do not handle --cluster-yes for cluster fix mode 2020-05-22 12:37:49 +02:00
Benjamin Sergeant 57b4fb0d84 fix typo ... 2020-05-22 12:37:49 +02:00
Benjamin Sergeant 29f25e4111 Redis-cli 6.0.1 --cluster-yes doesn't work (fix #7246)
This make it so that all prompts for all redis-cli --cluster commands are automatically answered with a yes.
2020-05-22 12:37:49 +02:00
Oran Agra 00d8b92b89 fix valgrind test failure in replication test
in b4416280c i added more keys to that test to make it run longer
but in valgrind this now means the test times out, give valgrind more
time.
2020-05-22 12:37:49 +02:00
Oran Agra 5e17e6276c add regression test for the race in #7205
with the original version of 6.0.0, this test detects an excessive full
sync.
with the fix in 1a7cd2c0e, this test detects memory corruption,
especially when using libc allocator with or without valgrind.
2020-05-22 12:37:49 +02:00
antirez 96e7c011e2 Improve the PSYNC2 test reliability. 2020-05-22 12:37:49 +02:00
John Sully c505df8830 Merge tag '6.0.1' into unstable
Redis 6.0.1.
2020-05-21 23:41:36 -04:00
John Sully f214ada545 Missing noexcept 2020-05-21 23:25:50 -04:00
John Sully c10aa3b8b0 build failure on macos 2020-05-21 23:24:34 -04:00
John Sully 6311d709c3 Fix bad merge in CI.yml 2020-05-21 22:09:06 -04:00
John Sully 8ab77465db Merge commit 'bec68bff2978de88b23140f41b6a943162333b11' into unstable 2020-05-21 19:37:42 -04:00
John Sully 201bf73ef7 Merge commit '414debfd04d21c0b19d5f6bb5249398d83670f3d' into unstable 2020-05-21 18:57:28 -04:00
John Sully 49784c8122 Merge commit 'cbcd07777dc569618a34f59e5fd0de53178f4f1d' into unstable 2020-05-21 18:48:49 -04:00
John Sully 3e03f308b5 Merge commit '5719b3054a534e62c25ae97680ecd4f7238ba484' into unstable 2020-05-21 17:55:09 -04:00
John Sully 95cecb0229 Merge commit '026cc11b056f063631d990f1a9db45b4e583974e' into unstable 2020-05-21 17:45:15 -04:00
John Sully 7676f5b15f Merge commit '024c380b9da02bc4112822c0f5f9ac1388b4205b' into unstable 2020-05-21 17:36:53 -04:00
John Sully d5b057534a Merge commit 'eba28e2cea0b2632cf751426ada02adf24f273db' into unstable 2020-05-21 17:32:53 -04:00
John Sully 7a15f6dfc7 Merge commit 'c35a53169ffdbdf73f91224b6e62f6129fb9a838' into unstable 2020-05-21 17:30:56 -04:00
John Sully 712d825c23 Override missing new, fix new/delete mismatch 2020-05-20 15:57:08 -04:00
antirez 7803b1148d Redis 6.0.3. 2020-05-16 18:11:58 +02:00
antirez 1eab62f7e5 Remove the client from CLOSE_ASAP list before caching the master.
This was broken in 1a7cd2c: we identified a crash in the CI, what
was happening before the fix should be like that:

1. The client gets in the async free list.
2. However freeClient() gets called again against the same client
   which is a master.
3. The client arrived in freeClient() with the CLOSE_ASAP flag set.
4. The master gets cached, but NOT removed from the CLOSE_ASAP linked
   list.
5. The master client that was cached was immediately removed since it
   was still in the list.
6. Redis accessed a freed cached master.

This is how the crash looked like:

=== REDIS BUG REPORT START: Cut & paste starting from here ===
1092:S 16 May 2020 11:44:09.731 # Redis 999.999.999 crashed by signal: 11
1092:S 16 May 2020 11:44:09.731 # Crashed running the instruction at: 0x447e18
1092:S 16 May 2020 11:44:09.731 # Accessing address: 0xffffffffffffffff
1092:S 16 May 2020 11:44:09.731 # Failed assertion:  (:0)

------ STACK TRACE ------
EIP:
src/redis-server 127.0.0.1:21300(readQueryFromClient+0x48)[0x447e18]

And the 0xffff address access likely comes from accessing an SDS that is
set to NULL (we go -1 offset to read the header).
2020-05-16 18:04:17 +02:00
antirez ae306a3df6 Redis 6.0.2. 2020-05-15 22:29:52 +02:00
Yossi Gottlieb 16ba33c05b TLS: Fix test failures on recent Debian/Ubuntu.
Seems like on some systems choosing specific TLS v1/v1.1 versions no
longer works as expected. Test is reduced for v1.2 now which is still
good enough to test the mechansim, and matters most anyway.
2020-05-15 22:23:24 +02:00
Yossi Gottlieb 77ae66930c TLS: Add crypto locks for older OpenSSL support.
This is really required only for older OpenSSL versions.

Also, at the moment Redis does not use OpenSSL from multiple threads so
this will only be useful if modules end up doing that.
2020-05-15 22:23:24 +02:00
David Carlier 389697988a NetBSD build update.
This platform supports CPU affinity (but not OpenBSD).
2020-05-15 22:23:24 +02:00
Madelyn Olson 2435341d74 Added a refcount on timer events to prevent deletion of recursive timer calls 2020-05-15 22:23:24 +02:00
antirez 80c906bd30 Cache master without checking of deferred close flags.
The context is issue #7205: since the introduction of threaded I/O we close
clients asynchronously by default from readQueryFromClient(). So we
should no longer prevent the caching of the master client, to later
PSYNC incrementally, if such flags are set. However we also don't want
the master client to be cached with such flags (would be closed
immediately after being restored). And yet we want a way to understand
if a master was closed because of a protocol error, and in that case
prevent the caching.
2020-05-15 22:23:24 +02:00
John Sully 4073758874 Remove debugging logs 2020-05-14 22:18:29 -04:00
antirez 74249be4a2 Track events processed while blocked globally.
Related to #7234.
2020-05-14 11:29:43 +02:00
antirez 8bf660af90 Some rework of #7234. 2020-05-14 11:29:43 +02:00
Oran Agra 9da134cd88 fix redis 6.0 not freeing closed connections during loading.
This bug was introduced by a recent change in which readQueryFromClient
is using freeClientAsync, and despite the fact that now
freeClientsInAsyncFreeQueue is in beforeSleep, that's not enough since
it's not called during loading in processEventsWhileBlocked.
furthermore, afterSleep was called in that case but beforeSleep wasn't.

This bug also caused slowness sine the level-triggered mode of epoll
kept signaling these connections as readable causing us to keep doing
connRead again and again for ll of these, which keep accumulating.

now both before and after sleep are called, but not all of their actions
are performed during loading, some are only reserved for the main loop.

fixes issue #7215
2020-05-14 11:29:43 +02:00
antirez f7f219a137 Regression test for #7249. 2020-05-14 11:29:43 +02:00
antirez 693629585b rax.c updated from upstream antirez/rax. 2020-05-14 11:29:43 +02:00
antirez e3b5648df0 Tracking: send eviction messages when evicting entries.
A fix for #7249.
2020-05-14 11:29:43 +02:00
Oran Agra 5c41802d55 fix unstable replication test
this test which has coverage for varoius flows of diskless master was
failing randomly from time to time.

the failure was:
[err]: diskless all replicas drop during rdb pipe in tests/integration/replication.tcl
log message of '*Diskless rdb transfer, last replica dropped, killing fork child*' not found

what seemed to have happened is that the master didn't detect that all
replicas dropped by the time the replication ended, it thought that one
replica is still connected.

now the test takes a few seconds longer but it seems stable.
2020-05-14 11:29:43 +02:00
John 3543a3c763 more reliability fixes for multimaster 2020-05-11 05:38:21 -04:00
John 3122912920 Make multimaster tests more reliable 2020-05-11 05:23:47 -04:00
John Sully f11fdf6717 Update access LFU/LRU access times when overwriting key 2020-05-11 00:42:46 -04:00
John Sully 572bcca9c2 We should still send replies even if an async close is pending 2020-05-10 23:56:29 -04:00
John Sully 10fa064109 Fix race on clients_to_close 2020-05-10 23:14:15 -04:00
John Sully 7cd779d304 Fix crash in client unblock command 2020-05-10 22:08:50 -04:00
John Sully 628d168049 Fix crash during hook module test 2020-05-10 21:57:16 -04:00
John Sully 37423757b5 Fix crash in module tests 2020-05-10 21:06:38 -04:00
John Sully 58752bb8fe Run sentinel and module tests under CI 2020-05-10 18:05:05 -04:00
John Sully 7e7f70ff3e add help for hrename 2020-05-10 17:41:01 -04:00
John Sully f77c227b2d hrename tests 2020-05-10 17:14:44 -04:00
John Sully 21d842b0b0 Implement keydb.hrename command 2020-05-10 17:04:22 -04:00
ShooterIT a23cdbb94b Redis Benchmark: Fix coredump because of double free 2020-05-09 11:14:58 +02:00
antirez 1276058ea8 Cluster: clarify we always resolve the sender. 2020-05-09 11:14:58 +02:00
antirez 002fcde3da Cluster: refactor ping/data delay handling. 2020-05-09 11:14:58 +02:00
antirez 960186a71f Cluster: introduce data_received field.
We want to send pings and pongs at specific intervals, since our packets
also contain information about the configuration of the cluster and are
used for gossip. However since our cluster bus is used in a mixed way
for data (such as Pub/Sub or modules cluster messages) and metadata,
sometimes a very busy channel may delay the reception of pong packets.
So after discussing it in #7216, this commit introduces a new field that
is not exposed in the cluster, is only an internal information about
the last time we received any data from a given node: we use this field
in order to avoid detecting failures, claiming data reception of new
data from the node is a proof of liveness.
2020-05-09 11:14:58 +02:00
antirez cb683a84f7 Don't propagate spurious MULTI on DEBUG LOADAOF. 2020-05-08 10:37:36 +02:00
antirez 3672875b4c stringmatchlen() should not expect null terminated strings. 2020-05-08 10:37:36 +02:00
antirez 84d9766d6a Dump recent backlog on master query generating errors. 2020-05-08 10:37:36 +02:00
Brad Dunbar 24e12641d5 Remove unreachable branch. 2020-05-08 10:37:36 +02:00
Titouan Christophe ec1e106ec5 make struct user anonymous (only typedefed)
This works because this struct is never referenced by its name,
but always by its type.

This prevents a conflict with struct user from <sys/user.h>
when compiling against uclibc.

Signed-off-by: Titouan Christophe <titouan.christophe@railnova.eu>
2020-05-08 10:37:36 +02:00
hwware c7edffbd56 add jemalloc-bg-thread config in redis conf 2020-05-08 10:37:36 +02:00
antirez e48c37316e Test: --dont-clean should do first cleanup. 2020-05-08 10:37:36 +02:00
hwware 8a9c84f4a5 add include guard for lolwut.h 2020-05-08 10:37:36 +02:00
Benjamin Sergeant 1e561cfaaf Add --user argument to redis-benchmark.c (ACL) 2020-05-08 10:37:36 +02:00
antirez d1af82a883 Drop not needed part from #7194. 2020-05-08 10:37:36 +02:00
Muhammad Zahalqa 897a360d06 Fix compiler warnings on function rev(unsigned long) 2020-05-08 10:37:36 +02:00
Guy Benoish 3a441c7d95 XPENDING should not update consumer's seen-time
Same goes for XGROUP DELCONSUMER (But in this case, it doesn't
have any visible effect)
2020-05-08 10:37:35 +02:00
Deliang Yang c57d9146f4 reformat code 2020-05-08 10:37:35 +02:00
antirez ac316d8cc5 Move CRC64 initialization in main(). 2020-05-08 10:37:35 +02:00
Oran Agra 75addb4fe2 optimize memory usage of deferred replies - fixed
When deffered reply is added the previous reply node cannot be used so
all the extra space we allocated in it is wasted. in case someone uses
deffered replies in a loop, each time adding a small reply, each of
these reply nodes (the small string reply) would have consumed a 16k
block.
now when we add anther diferred reply node, we trim the unused portion
of the previous reply block.

see #7123

cherry picked from commit fb732f7a94
with fix to handle a crash with LIBC allocator, which apparently can
return the same pointer despite changing it's size.
i.e. shrinking an allocation of 16k into 56 bytes without changing the
pointer.
2020-05-08 10:37:35 +02:00
Oran Agra 3d3861dd88 add daily github actions with libc malloc and valgrind
* fix memlry leaks with diskless replica short read.
* fix a few timing issues with valgrind runs
* fix issue with valgrind and watchdog schedule signal

about the valgrind WD issue:
the stack trace test in logging.tcl, has issues with valgrind:
==28808== Can't extend stack to 0x1ffeffdb38 during signal delivery for thread 1:
==28808==   too small or bad protection modes

it seems to be some valgrind bug with SA_ONSTACK.
SA_ONSTACK seems unneeded since WD is not recursive (SA_NODEFER was removed),
also, not sure if it's even valid without a call to sigaltstack()
2020-05-08 10:37:35 +02:00
antirez fc7bc32045 Fix CRC64 initialization outside the Redis server itself. 2020-05-08 10:37:35 +02:00
hwware a6e55c096d Client Side Caching: Add Tracking Prefix Number Stats in Server Info 2020-05-08 10:37:35 +02:00
antirez b062fd5235 Fix NetBSD build by fixing redis_set_thread_title() support.
See #7188.
2020-05-08 10:37:35 +02:00
antirez 4efb25d9c0 Rework a bit the documentation for CPU pinning. 2020-05-08 10:37:35 +02:00
zhenwei pi d6436eb7cf Support setcpuaffinity on linux/bsd
Currently, there are several types of threads/child processes of a
redis server. Sometimes we need deeply optimise the performance of
redis, so we would like to isolate threads/processes.

There were some discussion about cpu affinity cases in the issue:
https://github.com/antirez/redis/issues/2863

So implement cpu affinity setting by redis.conf in this patch, then
we can config server_cpulist/bio_cpulist/aof_rewrite_cpulist/
bgsave_cpulist by cpu list.

Examples of cpulist in redis.conf:
server_cpulist 0-7:2      means cpu affinity 0,2,4,6
bio_cpulist 1,3           means cpu affinity 1,3
aof_rewrite_cpulist 8-11  means cpu affinity 8,9,10,11
bgsave_cpulist 1,10-11    means cpu affinity 1,10,11

Test on linux/freebsd, both work fine.

Signed-off-by: zhenwei pi <pizhenwei@bytedance.com>
2020-05-08 10:37:35 +02:00
John Sully 86f9ed6248 Disable cluster tests
They are unreliable on slow hardware.
2020-05-07 01:49:04 -04:00
antirez 47ff13637e Redis 6.0.1. 2020-05-02 00:10:20 +02:00
antirez db73d0998f Cast printf() argument to the format specifier.
We could use uint64_t specific macros, but after all it's simpler to
just use an obvious equivalent type plus casting: this will be a no op
and is simpler than fixed size types printf macros.
2020-05-02 00:04:19 +02:00
antirez 7c0fe72714 Revert "optimize memory usage of deferred replies"
This reverts commit fb732f7a94.
2020-05-02 00:04:19 +02:00
antirez 8fe25edc75 Save a call to stopThreadedIOIfNeeded() for the base case.
Probably no performance changes, but the code should be trivial to
read as in "No threading? Use the normal function and return".
2020-05-02 00:04:19 +02:00
Diab Neiroukh dcd7b684bb Clang requires libatomic to be linked via LDFLAGS
Clang requires the libatomic to be linked via LDFLAGS or else the linker
will fail to recognise __atomic macros.

An error as a result of not passing -latomic to the linker can be seen
below:

ld.lld: error: undefined symbol: __atomic_fetch_add_2
>>> referenced by sds.c:185
>>>               lto.tmp:(sdsdupshared)
clang-10: error: linker command failed with exit code 1 (use -v to see invocation)
make[1]: *** [Makefile:276: keydb-server] Error 1
2020-04-30 15:06:10 -04:00
antirez 17dfd7cabb Redis 6.0.0 GA. 2020-04-30 15:04:41 +02:00
antirez 1f9b82bd56 Update help.h again before Redis 6 GA. 2020-04-30 13:43:58 +02:00
antirez 3fcffe7d04 redis-cli: fix hints with subcommands. 2020-04-30 13:43:58 +02:00
antirez 455d8a05c1 redis-cli command help updated. 2020-04-30 13:02:59 +02:00
zhaozhao.zz 70287bbc94 lazyfree & eviction: record latency generated by lazyfree eviction
1. add eviction-lazyfree monitor
2. put eviction-del & eviction-lazyfree into eviction-cycle
   that means eviction-cycle contains all the latency in
   the eviction cycle including del and lazyfree
3. use getMaxmemoryState to check if we can break in lazyfree-evict
2020-04-30 13:02:59 +02:00
antirez 7be21139aa MIGRATE AUTH2 for ACL support. 2020-04-30 13:02:59 +02:00
antirez e1ee1a49d6 CLIENT KILL USER <username>. 2020-04-30 13:02:59 +02:00
antirez d56f058c08 Fix tracking table max keys option in redis.conf. 2020-04-30 13:02:58 +02:00
antirez 96dd5fc932 redis-cli: safer cluster fix with unreachalbe masters. 2020-04-30 13:02:58 +02:00
antirez 5b59d9c5d7 redis-cli: simplify cluster nodes coverage display. 2020-04-30 13:02:58 +02:00
antirez c163d4addc redis-cli: try to make clusterManagerFixOpenSlot() more readable.
Also improve the message to make clear that there is no *clear* owner,
not that there is no owner at all.
2020-04-30 13:02:58 +02:00
Guy Benoish aab74b7151 XINFO STREAM FULL should have a default COUNT of 10 2020-04-30 13:02:58 +02:00
antirez 606134f9de Comment clearly why we moved some code in #6623. 2020-04-30 13:02:58 +02:00
srzhao ee627bb667 fix pipelined WAIT performance issue.
If client gets blocked again in `processUnblockedClients`, redis will not send
`REPLCONF GETACK *` to slaves untill next eventloop, so the client will be
blocked for 100ms by default(10hz) if no other file event fired.

move server.get_ack_from_slaves sinppet after `processUnblockedClients`, so
that both the first WAIT command that puts client in blocked context and the
following WAIT command processed in processUnblockedClients would trigger
redis-sever to send `REPLCONF GETACK *`, so that the eventloop would get
`REPLCONG ACK <reploffset>` from slaves and unblocked ASAP.
2020-04-30 13:02:58 +02:00
antirez 47b8a7f9b8 Fix create-cluster BIN_PATH. 2020-04-30 13:02:58 +02:00
Guy Benoish 6c0bc608a1 Extend XINFO STREAM output
Introducing XINFO STREAM <key> FULL
2020-04-30 13:02:58 +02:00
hwware 5bfc18950a Fix not used marco in cluster.c 2020-04-30 13:02:58 +02:00
Itamar Haber 56d628f851 Update create-cluster 2020-04-30 13:02:58 +02:00
Itamar Haber cac9d7cf7e Adds BIN_PATH to create-cluster
Allows for setting the binaries path if used outside the upstream repo.

Also documents `call` in usage clause (TODO: port to
`redis-cli --cluster call` or just deprecate it).
2020-04-30 13:02:58 +02:00
John Sully f715d0b860 cluster tests should be a named task 2020-04-29 18:43:55 -04:00
John Sully 98d690b849 Run cluster tests as part of CI 2020-04-29 18:41:51 -04:00
John Sully 45535e8a63 use serverAssert() instead of assert() to get callstacks in fastlock 2020-04-28 22:41:07 -04:00
John Sully c78895aaea Merge branch 'unstable' of https://github.com/JohnSully/KeyDB into unstable 2020-04-28 22:24:45 -04:00
Oran Agra b712fba17c hickup, re-fix dictEncObjKeyCompare
come to think of it, in theory (not in practice), getDecodedObject can
return the same original object with refcount incremented, so the
pointer comparision in the previous commit was invalid.
so now instead of checking the encoding, we explicitly check the
refcount.
2020-04-28 11:20:15 +02:00
Oran Agra ea63aea72d fix loading race in psync2 tests 2020-04-28 11:20:15 +02:00
antirez 64e588bfab Rework comment in dictEncObjKeyCompare(). 2020-04-28 11:20:15 +02:00
Oran Agra 0d1e8c93bf allow dictFind using static robj
since the recent addition of OBJ_STATIC_REFCOUNT and the assertion in
incrRefCount it is now impossible to use dictFind using a static robj,
because dictEncObjKeyCompare will call getDecodedObject which tries to
increment the refcount just in order to decrement it later.
2020-04-28 11:20:15 +02:00
Madelyn Olson a1bed447b3 Added crcspeed library 2020-04-28 11:20:15 +02:00
Madelyn Olson a75fa3aad1 Made crc64 test consistent 2020-04-28 11:20:15 +02:00
Madelyn Olson 52c75e9db1 Implemented CRC64 based on slice by 4 2020-04-28 11:20:15 +02:00
Oran Agra 8110ba8880 optimize memory usage of deferred replies
When deffered reply is added the previous reply node cannot be used so
all the extra space we allocated in it is wasted. in case someone uses
deffered replies in a loop, each time adding a small reply, each of
these reply nodes (the small string reply) would have consumed a 16k
block.
now when we add anther diferred reply node, we trim the unused portion
of the previous reply block.

see #7123
2020-04-27 16:46:14 +02:00
Oran Agra e4d2bb62b2 Keep track of meaningful replication offset in replicas too
Now both master and replicas keep track of the last replication offset
that contains meaningful data (ignoring the tailing pings), and both
trim that tail from the replication backlog, and the offset with which
they try to use for psync.

the implication is that if someone missed some pings, or even have
excessive pings that the promoted replica has, it'll still be able to
psync (avoid full sync).

the downside (which was already committed) is that replicas running old
code may fail to psync, since the promoted replica trims pings form it's
backlog.

This commit adds a test that reproduces several cases of promotions and
demotions with stale and non-stale pings

Background:
The mearningful offset on the master was added recently to solve a problem were
the master is left all alone, injecting PINGs into it's backlog when no one is
listening and then gets demoted and tries to replicate from a replica that didn't
have any of the PINGs (or at least not the last ones).

however, consider this case:
master A has two replicas (B and C) replicating directly from it.
there's no traffic at all, and also no network issues, just many pings in the
tail of the backlog. now B gets promoted, A becomes a replica of B, and C
remains a replica of A. when A gets demoted, it trims the pings from its
backlog, and successfully replicate from B. however, C is still aware of
these PINGs, when it'll disconnect and re-connect to A, it'll ask for something
that's not in the backlog anymore (since A trimmed the tail of it's backlog),
and be forced to do a full sync (something it didn't have to do before the
meaningful offset fix).

Besides that, the psync2 test was always failing randomly here and there, it
turns out the reason were PINGs. Investigating it shows the following scenario:

cycle 1: redis #1 is master, and all the rest are direct replicas of #1
cycle 2: redis #2 is promoted to master, #1 is a replica of #2 and #3 is replica of #1
now we see that when #1 is demoted it prints:
17339:S 21 Apr 2020 11:16:38.523 * Using the meaningful offset 3929963 instead of 3929977 to exclude the final PINGs (14 bytes difference)
17339:S 21 Apr 2020 11:16:39.391 * Trying a partial resynchronization (request e2b3f8817735fdfe5fa4626766daa938b61419e5:3929964).
17339:S 21 Apr 2020 11:16:39.392 * Successful partial resynchronization with master.
and when #3 connects to the demoted #2, #2 says:
17339:S 21 Apr 2020 11:16:40.084 * Partial resynchronization not accepted: Requested offset for secondary ID was 3929978, but I can reply up to 3929964

so the issue here is that the meaningful offset feature saved the day for the
demoted master (since it needs to sync from a replica that didn't get the last
ping), but it didn't help one of the other replicas which did get the last ping.
2020-04-27 15:52:49 +02:00
antirez fea9788cc4 Fix STRALGO command flags. 2020-04-27 15:52:49 +02:00
Dave-in-lafayette 2144047e14 fix for unintended crash during panic response
If redis crashes early, before lua is set up (like, if File Descriptor 0 is closed before exec), it will crash again trying to print memory statistics.
2020-04-27 15:52:49 +02:00
Guy Benoish 43329c9b64 Add the stream tag to XSETID tests 2020-04-27 15:52:49 +02:00
Dave-in-lafayette 1e17d3de79 fix for crash during panic before all threads are up
If there's a panic before all threads have been started (say, if file descriptor 0 is closed at exec), the panic response will crash here again.
2020-04-27 15:52:49 +02:00
John Sully d5ede50b04 shared pointer comparisons with other pointers 2020-04-24 22:20:26 -04:00
John Sully fab6132cb3 EMBSTR size is lower than it needs to be 2020-04-24 22:19:55 -04:00
antirez 3722f89f49 LCS -> STRALGO LCS.
STRALGO should be a container for mostly read-only string
algorithms in Redis. The algorithms should have two main
characteristics:

1. They should be non trivial to compute, and often not part of
programming language standard libraries.
2. They should be fast enough that it is a good idea to have optimized C
implementations.

Next thing I would love to see? A small strings compression algorithm.
2020-04-24 16:49:27 +02:00
antirez 373ae6061a Also use propagate() in streamPropagateGroupID(). 2020-04-24 10:15:04 +02:00
yanhui13 f03f1fad67 add tcl test for cluster slots 2020-04-24 10:15:04 +02:00
yanhui13 374ffdf1c1 optimize the output of cluster slots 2020-04-24 10:15:04 +02:00
antirez 4db38d2efd Minor aesthetic changes to #7135. 2020-04-24 10:15:04 +02:00
Valentino Geron f0a261448c XREADGROUP with NOACK should propagate only one XGROUP SETID command 2020-04-24 10:15:04 +02:00
antirez fbdef6a9bd ACL: re-enable command execution of disabled users.
After all I changed idea again: enabled/disabled should have a more
clear meaning, and it only means: you can't authenticate with such user
with new connections, however old connections continue to work as
expected.
2020-04-24 10:15:04 +02:00
antirez 05a41da75b getRandomBytes(): use HMAC-SHA256.
Now that we have an interface to use this API directly, via ACL GENPASS,
we are no longer sure what people could do with it. So why don't make it
a strong primitive exported by Redis in order to create unique IDs and
so forth?

The implementation was tested against the test vectors that can
be found in RFC4231.
2020-04-24 10:14:48 +02:00
antirez 345c3768de ACL GENPASS: take number of bits as argument. 2020-04-24 10:14:48 +02:00
antirez 639c8a1d91 ACL GENPASS: emit 256 bits instead of 128. 2020-04-24 10:14:48 +02:00
antirez 321acea038 ACL: deny commands execution of disabled users. 2020-04-24 10:14:48 +02:00
Theo Buehler b0920e6e86 TLS: Fix build with SSL_OP_NO_CLIENT_RENEGOTIATION
There is no ssl in this scope, so the build breaks.
All the other options are set directly on the ctx.
2020-04-24 10:14:48 +02:00
Yossi Gottlieb 149b658b56 TLS: Fix build on older verisons of OpenSSL. 2020-04-24 10:14:48 +02:00
antirez 06917e581c Tracking: test expired keys notifications. 2020-04-24 10:14:48 +02:00
antirez e434b2ce4f Tracking: NOLOOP tests. 2020-04-24 10:14:48 +02:00
antirez f3a1728873 Tracking: signal key as modified when evicting. 2020-04-24 10:14:48 +02:00
antirez e63bb7ec8c Tracking: NOLOOP further implementation and fixes. 2020-04-24 10:14:48 +02:00
antirez 6791ff0525 Tracking: NOLOOP internals implementation. 2020-04-24 10:14:48 +02:00
antirez 725b8cc680 Implement redis_set_thread_title for MacOS.
Strange enough, pthread_setname_np() produces a warning for not defined
function even if pthread is included. Moreover the MacOS documentation
claims the return value for the function is void, but actually is int.

Related to #7089.
2020-04-24 10:14:48 +02:00
zhenwei pi 3575b8706d Threaded IO: set thread name for redis-server
Set thread name for each thread of redis-server, this helps us to
monitor the utilization and optimise the performance.

And suggested-by Salvatore, implement this feature for multi
platforms. Currently support linux and bsd, ignore other OS.

An exmaple on Linux:
 # top -d 5 -p `pidof redis-server ` -H

    PID USER      PR  NI    VIRT    RES    SHR S %CPU %MEM     TIME+ COMMAND
3682671 root      20   0  227744   8248   3836 R 99.2  0.0   0:19.53 redis-server
3682677 root      20   0  227744   8248   3836 S 26.4  0.0   0:04.15 io_thd_3
3682675 root      20   0  227744   8248   3836 S 23.6  0.0   0:03.98 io_thd_1
3682676 root      20   0  227744   8248   3836 S 23.6  0.0   0:03.97 io_thd_2
3682672 root      20   0  227744   8248   3836 S  0.2  0.0   0:00.02 bio_close_file
3682673 root      20   0  227744   8248   3836 S  0.2  0.0   0:00.02 bio_aof_fsync
3682674 root      20   0  227744   8248   3836 S  0.0  0.0   0:00.00 bio_lazy_free
3682678 root      20   0  227744   8248   3836 S  0.0  0.0   0:00.00 jemalloc_bg_thd
3682682 root      20   0  227744   8248   3836 S  0.0  0.0   0:00.00 jemalloc_bg_thd
3682683 root      20   0  227744   8248   3836 S  0.0  0.0   0:00.00 jemalloc_bg_thd
3682684 root      20   0  227744   8248   3836 S  0.0  0.0   0:00.00 jemalloc_bg_thd
3682685 root      20   0  227744   8248   3836 S  0.0  0.0   0:00.00 jemalloc_bg_thd
3682687 root      20   0  227744   8248   3836 S  0.0  0.0   0:00.00 jemalloc_bg_thd

Another exmaple on FreeBSD-12.1:
  PID USERNAME    PRI NICE   SIZE    RES STATE    C   TIME    WCPU COMMAND
 5212 root        100    0    48M  7280K CPU2     2   0:26  99.52% redis-server{redis-server}
 5212 root         38    0    48M  7280K umtxn    4   0:06  26.94% redis-server{io_thd_3}
 5212 root         36    0    48M  7280K umtxn    6   0:06  26.84% redis-server{io_thd_1}
 5212 root         39    0    48M  7280K umtxn    1   0:06  25.30% redis-server{io_thd_2}
 5212 root         20    0    48M  7280K uwait    3   0:00   0.00% redis-server{redis-server}
 5212 root         21    0    48M  7280K uwait    2   0:00   0.00% redis-server{bio_close_file}
 5212 root         21    0    48M  7280K uwait    3   0:00   0.00% redis-server{bio_aof_fsync}
 5212 root         21    0    48M  7280K uwait    0   0:00   0.00% redis-server{bio_lazy_free}

Signed-off-by: zhenwei pi <pizhenwei@bytedance.com>
2020-04-24 10:14:48 +02:00
antirez a76c67578c Sentinel: small refactoring of sentinelCollectTerminatedScripts().
Related to #7113.
2020-04-24 10:14:48 +02:00
omg-by 3a27064c41 fix(sentinel): sentinel.running_scripts will always increase more times and not reset
when trigger a always fail scripts, sentinel.running_scripts will increase ten times, however it
only decrease one times onretry the maximum. and it will't reset, when it become
SENTINEL_SCRIPT_MAX_RUNNING, sentinel don't trigger scripts.
2020-04-24 10:14:48 +02:00
John Sully e0a8709c09 Default yes in apt install 2020-04-22 02:17:28 -04:00
John Sully 17795fd5ce Run arm CI builds 2020-04-22 02:13:02 -04:00
John Sully d6738ffa5d MacOS Build Break Fix 2020-04-22 01:21:35 -04:00
antirez 5c4c73e2c2 A few comments and name changes for #7103. 2020-04-17 13:02:40 +02:00
Oran Agra 6148f94939 testsuite run the defrag latency test solo
this test is time sensitive and it sometimes fail to pass below the
latency threshold, even on strong machines.

this test was the reson we're running just 2 parallel tests in the
github actions CI, revering this.
2020-04-17 13:02:40 +02:00
Jamie Scott 51d3012d48 Adding acllog-max-len to Redis.conf
While playing with ACLs I noticed that acllog-max-len wasn't in the redis.conf, but was a supported config. 

This PR documents and adds the directive to the redis.conf file.
2020-04-17 13:02:40 +02:00
antirez c39f16c422 Fix XCLAIM propagation in AOF/replicas for blocking XREADGROUP.
See issue #7105.
2020-04-17 12:40:54 +02:00
John Sully e72642667f Merge branch 'unstable' of https://github.com/JohnSully/KeyDB into unstable 2020-04-16 20:12:14 -04:00
John Sully 3785b1efd4 Fix some clang warnings 2020-04-16 20:12:05 -04:00
John Sully 7aeae62f84 Always link libatomic 2020-04-16 20:01:24 -04:00
John Sully e5e33f8cb7 Fix warning in cluster.cpp 2020-04-16 20:01:13 -04:00
John Sully a836d5947d quiet TSAN in serverLog 2020-04-16 17:06:25 -04:00
antirez 7cf0a77d59 Redis 6.0-RC4. 2020-04-16 16:18:02 +02:00
antirez 9f594e2432 Update SDS to latest version. 2020-04-16 16:09:06 +02:00
antirez 48781dd95c RESP3: fix HELLO map len in Sentinel mode.
See #6160.
2020-04-16 11:22:18 +02:00
John Sully 7cdd5d9e41 Update Readme to use github CI badge 2020-04-15 23:21:10 -04:00
John Sully 3f5d3fec36 update TLS readme for KeyDB 2020-04-15 23:19:05 -04:00
John Sully 90094a2dca Deprecate travis and finish setting up github CI 2020-04-15 23:04:04 -04:00
John Sully cc9924ffa6 Merge branch 'unstable' into redis_6_merge 2020-04-15 23:00:13 -04:00
John Sully 5e6b4ab99e OS X build break 2020-04-15 22:43:40 -04:00
John Sully 2d6d374162 more dependencies for CI 2020-04-15 22:39:47 -04:00
John Sully 3137d4f35c Add CI dependency 2020-04-15 22:36:16 -04:00
John Sully 6ff9d23fd4 During AOF reload we can erroneously read incorrect aof_state values, so this variable must be read with the global lock acquired 2020-04-15 22:30:19 -04:00
John Sully d4939c838b Quiet test only ASAN fd race 2020-04-15 22:27:45 -04:00
John Sully cd306f1d23 Run all KeyDB instances in testmode during tests 2020-04-15 22:27:04 -04:00
John Sully 5fd07e0889 Fix race in sendBulk 2020-04-15 22:26:00 -04:00
John Sully b0796ff5fd Convert variables accessed outside lock to atomics 2020-04-15 22:25:17 -04:00
John Sully 9dfa074cd6 ASAN races in leak checker 2020-04-15 22:24:12 -04:00
John Sully 033761c5f9 Multithreading reliability, force single thread for test relying on internal behavior 2020-04-15 20:52:25 -04:00
John Sully 2926ac494e rename to KeyDB (merge) 2020-04-15 16:44:06 -04:00
John Sully 06143c2e8c Log stdout and stderr for cluster tests 2020-04-15 16:35:24 -04:00
John Sully f00d28fdab Fix TSAN race 2020-04-15 16:34:38 -04:00
John Sully 4705f29e2f Fix incorrect cluster slot tracking (regression from merge) 2020-04-15 16:34:07 -04:00
antirez 371ab0cff8 Don't allow empty spaces in ACL usernames.
Fixes issue #6418.
2020-04-15 18:40:11 +02:00
antirez b86140ac5f Don't allow empty spaces in ACL key patterns.
Fixes issue #6418.
2020-04-15 18:40:11 +02:00
antirez bec68bff29 Use the special static refcount for stack objects. 2020-04-15 16:03:16 +02:00
antirez 414debfd04 Speedup: unblock clients on keys in O(1).
See #7071.
2020-04-15 16:03:16 +02:00
liumiuyong a7ee3c3e77 FIX: truncate max/min longitude,latitude related geo_point (ex: {180, 85.05112878} ) 2020-04-15 16:03:16 +02:00
antirez 0f239e51b0 RDB: refactor some RDB loading code into dbAddRDBLoad(). 2020-04-15 16:03:16 +02:00
Guy Benoish e5b9eb8171 Typo in getTimeoutFromObjectOrReply's error reply 2020-04-15 16:03:16 +02:00
antirez f855db61b0 incrRefCount(): abort on statically allocated object. 2020-04-15 16:03:16 +02:00
antirez 0f31bb5c1a Fix HELLO reply in Sentinel mode, see #6160. 2020-04-15 16:03:16 +02:00
antirez 23094ba015 More powerful DEBUG RELOAD.
Related to #3243.
2020-04-15 16:03:16 +02:00
hwware b92d9a895f fix spelling in acl.c 2020-04-15 16:03:16 +02:00
antirez 8161a7a3ef RDB: clarify a condition in rdbLoadRio(). 2020-04-15 16:03:16 +02:00
antirez 8f896e57ae Fix zsetAdd() top comment spelling. 2020-04-15 16:03:16 +02:00
antirez 61b153073f RDB: load files faster avoiding useless free+realloc.
Reloading of the RDB generated by

    DEBUG POPULATE 5000000
    SAVE

is now 25% faster.

This commit also prepares the ability to have more flexibility when
loading stuff from the RDB, since we no longer use dbAdd() but can
control exactly how things are added in the database.
2020-04-15 16:03:16 +02:00
hayleeliu 8f5157058b fix spelling mistake in bitops.c 2020-04-15 16:03:16 +02:00
antirez ddeda9ceb7 Fix function names in zslDeleteNode() top comment. 2020-04-15 16:03:16 +02:00
antirez bde1f0a8e2 RESP3: change streams items from maps to arrays.
Streams items are similar to dictionaries, however they preserve both
the order, and allow for duplicated field names. So a map is not a
semantically sounding way to deal with this.

https://twitter.com/antirez/status/1248261087553880069
2020-04-15 16:03:16 +02:00
John Sully b2cb10de5f Merge tag '6.0-rc3' into redis_6_merge
Redis 6.0 RC3.
2020-04-14 22:56:19 -04:00
John Sully 1005a725d4 Fix merge issues and move timeout to C++ 2020-04-14 22:37:26 -04:00
John Sully 50768cd242 Merge commit 'c3b268a0bcded2d730790a772176483efb7c31fb' into redis_6_merge 2020-04-14 22:25:44 -04:00
John Sully cc3ebbe519 Merge commit '0f7dfc378ce5ab0524ac28b6142545bc41435386' into redis_6_merge 2020-04-14 22:19:29 -04:00
John Sully 8056e349e0 Merge commit '7c07841632500dfdff31f5a69e5bd140abea646c' into redis_6_merge 2020-04-14 21:26:51 -04:00
John Sully 0001f59c8d Merge commit 'b3e0305448422239ba3cb8d125681bccdb809cd3' into redis_6_merge 2020-04-14 21:16:38 -04:00
John Sully 2941792a85 Merge commit 'e628f94436f2ffb38dd6305f18956f309838a8d6' into redis_6_merge 2020-04-14 21:13:42 -04:00
John Sully 320bc3c032 Merge commit '76d57161d960f27ac92765118e3b80fef8847b8f' into redis_6_merge 2020-04-14 21:04:42 -04:00
John Sully 2825e51550 Merge commit 'e74e68c84f5eba8013769087c9a46cab811b8417' into redis_6_merge 2020-04-14 20:55:29 -04:00
John Sully cbdd47d056 Merge commit 'baaf869fc3e138cfcb4cfda09f09fd3c87c8f924' into redis_6_merge 2020-04-14 20:44:27 -04:00
John Sully 516c8c6e23 Merge commit 'be4bc1a5be26a7fde2fd05acd8187f5f0ed59f25' into redis_6_merge 2020-04-14 20:42:48 -04:00
John Sully ef1236b600 Merge commit '973297336fc05a601e17be70aba88e5dca6480ae' into redis_6_merge 2020-04-14 20:19:48 -04:00
John Sully 28cbed1d13 Merge commit '349aa24511253b3e323064b95cefb4414f31726c' into redis_6_merge 2020-04-14 20:09:53 -04:00
John Sully cde199a797 Merge commit 'df83892760d19b1b7e92d76e72daf4834ad2df6c' into redis_6_merge 2020-04-14 19:43:04 -04:00
John Sully 5589a0a69c Merge commit 'f42ce57d0f7eee6705fb0a81714cabe97a4f2c0a' into redis_6_merge 2020-04-14 19:22:44 -04:00
John Sully 09e8fb17cd Merge commit '51c1a9f8fbc12a9276489178242e498bb6ccbdba' into redis_6_merge 2020-04-14 18:44:42 -04:00
John Sully d71c4c2215 Merge commit '6fe55c2f299bbb306533af39cb28346bf5d473d3' into redis_6_merge 2020-04-14 18:30:26 -04:00
John Sully 6f621e6480 Merge commit '721a39ddff0eb9be822d27dba80267f6da149aff' into redis_6_merge 2020-04-14 18:29:03 -04:00
John Sully 8e33c2d301 Merge commit 'cbabf779c24fd107ca8e4fe6369e4677ad0eda10' into redis_6_merge 2020-04-14 18:05:10 -04:00
John Sully 52aca124c3 Merge commit 'ecd17e819cbb05110a49d81c6fda10e6770a7838' into redis_6_merge 2020-04-14 18:04:51 -04:00
antirez c7159d58f1 Jump to right label on AOF parsing error.
Related to #6054.
2020-04-14 18:03:22 -04:00
antirez 5d6e8fe3e3 PSYNC2: meaningful offset implemented.
A very commonly signaled operational problem with Redis master-replicas
sets is that, once the master becomes unavailable for some reason,
especially because of network problems, many times it wont be able to
perform a partial resynchronization with the new master, once it rejoins
the partition, for the following reason:

1. The master becomes isolated, however it keeps sending PINGs to the
replicas. Such PINGs will never be received since the link connection is
actually already severed.
2. On the other side, one of the replicas will turn into the new master,
setting its secondary replication ID offset to the one of the last
command received from the old master: this offset will not include the
PINGs sent by the master once the link was already disconnected.
3. When the master rejoins the partion and is turned into a replica, its
offset will be too advanced because of the PINGs, so a PSYNC will fail,
and a full synchronization will be required.

Related to issue #7002 and other discussion we had in the past around
this problem.
2020-04-14 17:56:09 -04:00
Oran Agra 99762dac23 reduce repeated calls to use_diskless_load
this function possibly iterates on the module list
2020-04-14 17:34:36 -04:00
John Sully 57956b3fd2 Fix merge conflicts from cherry-pick 2020-04-14 17:33:54 -04:00
antirez b323645227 Remove RDB files used for replication in persistence-less instances. 2020-04-14 17:27:05 -04:00
Johannes Truschnigg dd9502f373 Signal systemd readiness atfer Partial Resync
"Partial Resynchronization" is a special variant of replication success
that we have to tell systemd about if it is managing redis-server via a
Type=Notify service unit.
2020-04-14 17:22:30 -04:00
Guy Benoish aa8a3077a2 Diskless-load emptyDb-related fixes
1. Call emptyDb even in case of diskless-load: We want modules
   to get the same FLUSHDB event as disk-based replication.
2. Do not fire any module events when flushing the backups array.
3. Delete redundant call to signalFlushedDb (Called from emptyDb).
2020-04-14 17:21:10 -04:00
John Sully 8e81e5f29e Cache fake client in replicaFeedSlaves 2020-04-13 22:45:15 -04:00
John Sully 63dd1fb599 Don't do active replica work if we're not an active replica 2020-04-13 22:35:06 -04:00
John Sully 32561a9912 Optimize replicaFeedSlaves by removing use of snprintf 2020-04-13 22:11:34 -04:00
antirez cbcd07777d Fix ACL HELP table missing comma. 2020-04-08 10:56:31 +02:00
mymilkbottles 2437455f27 Judge the log level in advance 2020-04-07 16:53:13 +02:00
antirez 35c64b898c Speedup INFO by counting client memory incrementally.
Related to #5145.

Design note: clients may change type when they turn into replicas or are
moved into the Pub/Sub category and so forth. Moreover the recomputation
of the bytes used is problematic for obvious reasons: it changes
continuously, so as a conservative way to avoid accumulating errors,
each client remembers the contribution it gave to the sum, and removes
it when it is freed or before updating it with the new memory usage.
2020-04-07 16:53:13 +02:00
qetu3790 c3ac717487 fix comments about RESIZE DB opcode in rdb.c
fix comments about RESIZE DB opcode in rdb.c
2020-04-07 16:53:13 +02:00
antirez c8dbcff9db Clarify redis.conf comment about lazyfree-lazy-user-del. 2020-04-07 16:53:13 +02:00
zhaozhao.zz abd5156f25 lazyfree: add a new configuration lazyfree-lazy-user-del
Delete keys in async way when executing DEL command, if
lazyfree-lazy-user-del is yes.
2020-04-07 16:53:13 +02:00
antirez 5719b3054a LCS: more tests. 2020-04-07 16:52:57 +02:00
antirez c89e1f2937 LCS: allow KEYS / STRINGS to be anywhere.
Initially they needed to be at the end so that we could extend to N
strings in the future, but after further consideration I no longer
believe it's worth it.
2020-04-07 16:52:57 +02:00
antirez 0b16f8d444 LCS tests. 2020-04-07 16:52:57 +02:00
antirez 9254a805d7 LCS: get rid of STOREIDX option. Fix get keys helper. 2020-04-07 16:52:57 +02:00
antirez a4c4907035 LCS: fix stale comment. 2020-04-07 16:52:57 +02:00
antirez cb92c23de0 LCS: output LCS len as well in IDX mode. 2020-04-07 16:52:57 +02:00
antirez 56a52e804c LCS: MINMATCHLEN and WITHMATCHLEN options. 2020-04-07 16:52:57 +02:00
antirez ebb09a5c3b LCS: 7x speedup by accessing the array with better locality. 2020-04-07 16:52:57 +02:00
antirez a9f8a8cba0 LCS: implement KEYS option. 2020-04-07 16:52:57 +02:00
antirez 4aa24e62a4 LCS: other fixes to range emission. 2020-04-07 16:52:57 +02:00
antirez 2b67b6b87e LCS: fix emission of last range starting at index 0. 2020-04-07 16:52:57 +02:00
antirez 420aac7274 LCS: implement range indexes option. 2020-04-07 16:52:57 +02:00
antirez a518a9a766 LCS: initial functionality implemented. 2020-04-07 16:52:57 +02:00
srzhao 026cc11b05 Check OOM at script start to get stable lua OOM state.
Checking OOM by `getMaxMemoryState` inside script might get different result
with `freeMemoryIfNeededAndSafe` at script start, because lua stack and
arguments also consume memory.

This leads to memory `borderline` when memory grows near server.maxmemory:

- `freeMemoryIfNeededAndSafe` at script start detects no OOM, no memory freed
- `getMaxMemoryState` inside script detects OOM, script aborted

We solve this 'borderline' issue by saving OOM state at script start to get
stable lua OOM state.

related to issue #6565 and #5250.
2020-04-07 16:52:28 +02:00
Oran Agra 02b594f6ad diffrent fix for runtest --host --port 2020-04-07 16:52:28 +02:00
Guy Benoish f695d18306 Try to fix time-sensitive tests in blockonkey.tcl
There is an inherent race between the deferring client and the
"main" client of the test: While the deferring client issues a blocking
command, we can't know for sure that by the time the "main" client
tries to issue another command (Usually one that unblocks the deferring
client) the deferring client is even blocked...
For lack of a better choice this commit uses TCL's 'after' in order
to give some time for the deferring client to issues its blocking
command before the "main" client does its thing.
This problem probably exists in many other tests but this commit
tries to fix blockonkeys.tcl
2020-04-07 16:52:28 +02:00
Guy Benoish 0e42cfc365 Use __attribute__ only if __GNUC__ is defined 2020-04-07 16:52:28 +02:00
Guy Benoish 024c380b9d Fix no-negative-zero test 2020-04-07 16:52:04 +02:00
Guy Benoish eba28e2cea DEBUG OBJECT should pass keyname to module when loading 2020-04-07 16:52:04 +02:00
Guy Benoish c35a53169f streamReplyWithRange: Redundant XSETIDs to replica
propagate_last_id is declared outside of the loop but used
only from within the loop. Once it's '1' it will never go
back to '0' and will replicate XSETID even for IDs that
don't actually change the last_id.
While not a serious bug (XSETID always used group->last_id
so there's no risk), it does causes redundant traffic
between master and its replicas
2020-04-07 16:52:04 +02:00
Guy Benoish 91ed9b3c47 Modules: Perform printf-like format checks in variadic API 2020-04-07 16:52:04 +02:00
Oran Agra a38ff404bd modules don't signalModifiedKey in setKey() since that's done (optionally) in RM_CloseKey 2020-04-07 16:52:04 +02:00
David Carlier 15c9e79a7d debug, dump registers on arm too. 2020-04-07 16:52:04 +02:00
antirez 6fe66e0969 Simplify comment in moduleTryServeClientBlockedOnKey(). 2020-04-07 16:52:04 +02:00
Valentino Geron 3e0d209625 XREAD and XREADGROUP should not be allowed from scripts when BLOCK option is being used 2020-04-07 16:52:04 +02:00
Oran Agra 814874d68b change CI to build and run the module api tests 2020-04-07 16:52:04 +02:00
hwware cd2b5df971 fix spelling in cluster.c 2020-04-07 16:52:04 +02:00
Guy Benoish 240094c9b0 Stale replica should allow MULTI/EXEC
Example: Client uses a pipe to send the following to a
stale replica:

MULTI
.. do something ...
DISCARD

The replica will reply the MUTLI with -MASTERDOWN and
execute the rest of the commands... A client using a
pipe might not be aware that MULTI failed until it's
too late.

I can't think of a reason why MULTI/EXEC/DISCARD should
not be executed on stale replicas...

Also, enable MULTI/EXEC/DISCARD during loading
2020-04-07 16:52:04 +02:00
Oran Agra 061616c1b1 fix possible warning on incomplete struct init 2020-04-07 16:52:04 +02:00
Valentino Geron 8cdc153f58 XACK should be executed in a "all or nothing" fashion.
First, we must parse the IDs, so that we abort ASAP.
The return value of this command cannot be an error if
the client successfully acknowledged some messages,
so it should be executed in a "all or nothing" fashion.
2020-04-07 16:52:04 +02:00
Xudong Zhang 209f3a1eba fix integer overflow 2020-04-07 16:52:04 +02:00
Guy Benoish 7764996bec Make sure Redis does not reply with negative zero 2020-04-07 16:52:04 +02:00
hwware b35407fa7c add check for not switching between optin optout mode directly 2020-04-07 16:52:04 +02:00
hwware 4395889c9e add check for not providing both optin optout flag 2020-04-07 16:52:04 +02:00
Guy Benoish 1907e0f18f PERSIST should notify a keyspace event 2020-04-07 16:52:04 +02:00
Guy Benoish 193fc241ca Fix memory corruption in moduleHandleBlockedClients
By using a "circular BRPOPLPUSH"-like scenario it was
possible the get the same client on db->blocking_keys
twice (See comment in moduleTryServeClientBlockedOnKey)

The fix was actually already implememnted in
moduleTryServeClientBlockedOnKey but it had a bug:
the funxction should return 0 or 1 (not OK or ERR)

Other changes:
1. Added two commands to blockonkeys.c test module (To
   reproduce the case described above)
2. Simplify blockonkeys.c in order to make testing easier
3. cast raxSize() to avoid warning with format spec
2020-04-07 16:52:03 +02:00
John Sully 3fdc8608f3 Fix failure to AUTH with master when masterauth is set last 2020-04-06 01:24:59 -04:00
John Sully 9e2e2067c6 Add the ability to set a starting core # when setting thread affinity 2020-04-04 22:58:17 -04:00
John Sully f112c77fcc Fix issue #164 2020-04-04 22:45:12 -04:00
John Sully 888d69a87a Role command protocol corruption with multiple masters 2020-04-04 22:32:15 -04:00
John Sully bd183cdee1 Fix subkey expires not replicating correctly, and AOF issues 2020-04-04 21:52:27 -04:00
antirez 957e917a84 Redis 6.0-RC3. 2020-03-31 17:56:04 +02:00
antirez ef1b1f01a8 cast raxSize() to avoid warning with format spec. 2020-03-31 17:41:23 +02:00
antirez 9f347fabba Minor changes to #7037. 2020-03-31 17:12:19 +02:00
Guy Benoish a509400d58 Modules: Test MULTI/EXEC replication of RM_Replicate
Makse sure call() doesn't wrap replicated commands with
a redundant MULTI/EXEC

Other, unrelated changes:
1. Formatting compiler warning in INFO CLIENTS
2. Use CLIENT_ID_AOF instead of UINT64_MAX
2020-03-31 17:12:19 +02:00
antirez c3b268a0bc timeout.c created: move client timeouts code there. 2020-03-31 16:57:20 +02:00
Oran Agra 0f7dfc378c AOFRW on an empty stream created with MKSTREAM loads badkly
the AOF will be loaded successfully, but the stream will be missing,
i.e inconsistencies with the original db.

this was because XADD with id of 0-0 would error.

add a test to reproduce.
2020-03-31 16:57:20 +02:00
Guy Benoish 805c8c94a9 RENAME can unblock XREADGROUP
Other changes:
Support stream in serverLogObjectDebugInfo
2020-03-31 16:57:20 +02:00
antirez 97b80b57e9 Fix the propagate Tcl test after module changes. 2020-03-31 16:57:20 +02:00
antirez 4f6b6b8008 Modify the propagate unit test to show more cases. 2020-03-31 16:57:20 +02:00
antirez 616b1cb7ac Fix module commands propagation double MULTI bug.
37a10cef introduced automatic wrapping of MULTI/EXEC for the
alsoPropagate API. However this collides with the built-in mechanism
already present in module.c. To avoid complex changes near Redis 6 GA
this commit introduces the ability to exclude call() MUTLI/EXEC wrapping
for also propagate in order to continue to use the old code paths in
module.c.
2020-03-31 16:57:20 +02:00
antirez 08fdef4bf6 Fix RM_Call() stale comment due to cut&paste. 2020-03-31 16:57:20 +02:00
OMG-By 26b79ca18c fix: dict.c->dictResize()->minimal type 2020-03-31 16:57:20 +02:00
zhaozhao.zz fa41863752 PSYNC2: reset backlog_idx and master_repl_offset correctly 2020-03-31 16:57:20 +02:00
antirez bbbc80acae Precise timeouts: reference client pointer directly. 2020-03-31 16:57:20 +02:00
antirez 67643ead69 Precise timeouts: cleaup the table on unblock.
Now that this mechanism is the sole one used for blocked clients
timeouts, it is more wise to cleanup the table when the client unblocks
for any reason. We use a flag: CLIENT_IN_TO_TABLE, in order to avoid a
radix tree lookup when the client was already removed from the table
because we processed it by scanning the radix tree.
2020-03-31 16:56:46 +02:00
antirez ad94066ec8 Precise timeouts: fix comments after functional change. 2020-03-31 16:56:46 +02:00
antirez a443ec2e9a Precise timeouts: use only radix tree for timeouts. 2020-03-31 16:56:46 +02:00
antirez 6862fd70da Precise timeouts: fast exit for clientsHandleShortTimeout(). 2020-03-31 16:56:46 +02:00
antirez 30f1df8c48 Precise timeouts: fix bugs in initial implementation. 2020-03-31 16:56:46 +02:00
antirez 7add0f2490 Precise timeouts: working initial implementation. 2020-03-31 16:56:46 +02:00
antirez 9d6d177994 Precise timeouts: refactor unblocking on timeout. 2020-03-31 16:56:46 +02:00
antirez 316a8f1545 PSYNC2: fix backlog_idx when adjusting for meaningful offset
See #7002.
2020-03-31 16:56:46 +02:00
伯成 11db53f875 Boost up performance for redis PUB-SUB patterns matching
If lots of clients PSUBSCRIBE to same patterns, multiple pattens matching will take place. This commit change it into just one single pattern matching by using a `dict *` to store the unique pattern and which clients subscribe to it.
2020-03-31 12:47:14 +02:00
John Sully 95b4f899c5 Merge branch 'redis_6_merge' of https://github.com/JohnSully/KeyDB into redis_6_merge 2020-03-30 00:52:43 -04:00
John Sully 2946155313 Merge branch 'unstable' into redis_6_merge 2020-03-30 00:52:12 -04:00
John Sully f2ca2aa1f7 Allow active replicas to connect to each other when replica-serve-stale-data is set 2020-03-30 00:51:40 -04:00
John Sully b20bf10fda Don't send build log to stderr 2020-03-28 23:36:57 -04:00
John Sully 208c15bc16 Merge branch 'unstable' into redis_6_merge 2020-03-27 14:50:37 -04:00
John Sully c3d3c82f3a Raspberry Pi fixes (compile and replication) 2020-03-27 12:46:25 -04:00
John Sully b22d9cc27d Fix merge issues 2020-03-25 22:26:27 -04:00
John Sully 908cf5042e Merge branch 'unstable' into redis_6_merge 2020-03-25 22:12:22 -04:00
John Sully 6ec1641294 Fix issue #143 2020-03-25 21:55:31 -04:00
John Sully 7b79ec360b Give a better error when handling std::terminate 2020-03-25 16:27:24 -04:00
John Sully 718aee242d Merge branch 'unstable' into redis_6_merge 2020-03-25 15:47:12 -04:00
John Sully 56fc6b7deb Add roundtrip test for subkey expires 2020-03-25 15:34:30 -04:00
antirez e257f121e1 PSYNC2: meaningful offset test. 2020-03-25 15:55:24 +01:00
antirez 5f72f69688 PSYNC2: meaningful offset implemented.
A very commonly signaled operational problem with Redis master-replicas
sets is that, once the master becomes unavailable for some reason,
especially because of network problems, many times it wont be able to
perform a partial resynchronization with the new master, once it rejoins
the partition, for the following reason:

1. The master becomes isolated, however it keeps sending PINGs to the
replicas. Such PINGs will never be received since the link connection is
actually already severed.
2. On the other side, one of the replicas will turn into the new master,
setting its secondary replication ID offset to the one of the last
command received from the old master: this offset will not include the
PINGs sent by the master once the link was already disconnected.
3. When the master rejoins the partion and is turned into a replica, its
offset will be too advanced because of the PINGs, so a PSYNC will fail,
and a full synchronization will be required.

Related to issue #7002 and other discussion we had in the past around
this problem.
2020-03-25 15:55:24 +01:00
antirez 8caa271476 Explain why we allow transactions in -BUSY state.
Related to #7022.
2020-03-25 15:55:24 +01:00
Oran Agra e43cd8316f MULTI/EXEC during LUA script timeout are messed up
Redis refusing to run MULTI or EXEC during script timeout may cause partial
transactions to run.

1) if the client sends MULTI+commands+EXEC in pipeline without waiting for
response, but these arrive to the shards partially while there's a busy script,
and partially after it eventually finishes: we'll end up running only part of
the transaction (since multi was ignored, and exec would fail).

2) similar to the above if EXEC arrives during busy script, it'll be ignored and
the client state remains in a transaction.

the 3rd test which i added for a case where MULTI and EXEC are ok, and
only the body arrives during busy script was already handled correctly
since processCommand calls flagTransaction
2020-03-25 15:55:24 +01:00
antirez 34b8983220 Improve comments of replicationCacheMasterUsingMyself(). 2020-03-25 15:55:24 +01:00
antirez 70a98a43ea Fix BITFIELD_RO test. 2020-03-25 15:55:24 +01:00
antirez 8783304a2d Abort transactions after -READONLY error. Fix #7014. 2020-03-25 15:55:24 +01:00
antirez ec9cf002d5 Minor changes to BITFIELD_RO PR #6951. 2020-03-25 15:55:24 +01:00
bodong.ybd b3e4abf06e Added BITFIELD_RO variants for read-only operations. 2020-03-25 15:55:24 +01:00
antirez 50f8f9504b Modules: updated function doc after #7003. 2020-03-25 15:55:24 +01:00
Guy Benoish f2f3dc5e73 Allow RM_GetContextFlags to work with ctx==NULL 2020-03-25 15:55:24 +01:00
hwware eb80887936 fix potentical memory leak in redis-cli 2020-03-25 15:55:24 +01:00
Yossi Gottlieb cdcab0e820 Fix crashes related to failed/rejected accepts. 2020-03-25 15:55:24 +01:00
Yossi Gottlieb 50dcd9f96d Cluster: fix misleading accept errors. 2020-03-25 15:55:24 +01:00
Yossi Gottlieb 87dbd8f54c Conns: Fix connClose() / connAccept() behavior.
We assume accept handlers may choose to reject a connection and close
it, but connAccept() callers can't distinguish between this state and
other error states requiring connClose().

This makes it safe (and mandatory!) to always call connClose() if
connAccept() fails, and safe for accept handlers to close connections
(which will defer).
2020-03-25 15:55:24 +01:00
hwware 81e8686cc7 remove redundant Semicolon 2020-03-25 15:55:24 +01:00
hwware c7524a7e44 clean CLIENT_TRACKING_CACHING flag when disabled caching 2020-03-25 15:55:24 +01:00
hwware 2dd1ca6af0 add missing commands in cluster help 2020-03-25 15:55:24 +01:00
artix 95324b8190 Support Redis Cluster Proxy PROXY INFO command 2020-03-25 15:55:24 +01:00
zhaozhao.zz 7c07841632 Threaded IO: bugfix client kill may crash redis 2020-03-25 15:54:34 +01:00
zhaozhao.zz b3e0305448 Threaded IO: bugfix #6988 process events while blocked 2020-03-25 15:54:34 +01:00
antirez e628f94436 Restore newline at the end of redis-cli.c 2020-03-25 15:54:34 +01:00
bodong.ybd 76d57161d9 Fix bug of tcl test using external server 2020-03-25 15:54:34 +01:00
박승현 04c53fa145 Update redis.conf 2020-03-25 15:54:34 +01:00
zhaozhao.zz 9cc7038e54 Threaded IO: handle pending reads clients ASAP after event loop 2020-03-25 15:54:34 +01:00
chendianqiang 5d4c4df3ef use correct list for moduleUnregisterUsedAPI 2020-03-25 15:54:34 +01:00
fengpf 0e5820d893 fix comments in latency.c 2020-03-25 15:54:34 +01:00
WuYunlong 0578157d56 Fix master replica inconsistency for upgrading scenario.
Before this commit, when upgrading a replica, expired keys will not
be loaded, thus causing replica having less keys in db. To this point,
master and replica's keys is logically consistent. However, before
the keys in master and replica are physically consistent, that is,
they have the same dbsize, if master got a problem and the replica
got promoted and becomes new master of that partition, and master
updates a key which does not exist on master, but physically exists
on the old master(new replica), the old master would refuse to update
the key, thus causing master and replica data inconsistent.

How could this happen?
That's all because of the wrong judgement of roles while starting up
the server. We can not use server.masterhost to judge if the server
is master or replica, since it fails in cluster mode.

When we start the server, we load rdb and do want to load expired keys,
and do not want to have the ability to active expire keys, if it is
a replica.
2020-03-25 15:54:34 +01:00
antirez da8c7c49bf Example sentinel conf: document requirepass. 2020-03-25 15:54:34 +01:00
guodongxiaren da14982d1e string literal should be const char* 2020-03-25 15:54:34 +01:00
WuYunlong 299f1d0258 Add 14-consistency-check.tcl to prove there is a data consistency issue. 2020-03-25 15:54:34 +01:00
antirez bdb338cf77 Aesthetic changes in PR #6989. 2020-03-25 15:54:34 +01:00
Itamar Haber dc8885a1ca Adds keyspace notifications to migrate and restore 2020-03-25 15:54:34 +01:00
antirez 61b98f32a2 Regression test for #7011. 2020-03-25 15:54:34 +01:00
bodong.ybd bfb18e5519 Remove duplicate obj files in Makefile 2020-03-25 15:54:34 +01:00
antirez 34ea2f4e1a ACL: default user off should not allow automatic authentication.
This fixes issue #7011.
2020-03-25 15:54:34 +01:00
antirez cbbf9b3931 Sentinel: document auth-user directive. 2020-03-25 15:54:34 +01:00
antirez 9c2e42ddfc ACL: Make Redis 6 more backward compatible with requirepass.
Note that this as a side effect fixes Sentinel "requirepass" mode.
2020-03-25 15:54:34 +01:00
antirez d387f67dcb Sentinel: implement auth-user directive for ACLs. 2020-03-25 15:54:34 +01:00
John Sully 528a43bd6c Fix failure to load subkey expires 2020-03-25 01:38:58 -04:00
John Sully b357803362 Expire entry needs to be resorted after a subkey expires 2020-03-25 01:06:40 -04:00
John Sully 448808529a Unstable branch should always be version 0.0.0 2020-03-19 15:52:24 -04:00
John Sully a0bc208412 Merge branch 'RELEASE_5' into unstable 2020-03-19 15:38:16 -04:00
John Sully 6b0172b9bf Print stack traces of other threads in the deadlock detector 2020-03-19 15:37:49 -04:00
John Sully bc1eccb66d Log which thread a message came from 2020-03-19 15:37:24 -04:00
John Sully a9249d4a82 Fix lock inversion in processEventsWhileBlocked 2020-03-19 15:28:39 -04:00
John Sully 3c373494d6 Prevent dangling lock when we can't free the client 2020-03-19 14:34:23 -04:00
John Sully 40761e4031 Merge branch 'unstable' into redis_6_merge 2020-03-17 00:09:48 -04:00
John Sully 5aef606d3b Fix issue #153: Config get replicaof returns a corrupt response 2020-03-16 20:43:46 -04:00
John Sully 4671cd6002 Merge branch 'unstable' into redis_6_merge 2020-03-15 18:47:27 -04:00
John Sully 2ec0b2cd20 Handle HTTP error codes with MOTD 2020-03-15 18:47:01 -04:00
John Sully 07effa3925 Merge branch 'unstable' into redis_6_merge 2020-03-15 18:33:24 -04:00
John Sully cd76a71de3 Update MOTD URL 2020-03-15 18:32:05 -04:00
antirez 916dd79f19 Update linenoise. 2020-03-12 15:53:47 +01:00
lifubang c0c67c9be3 add askpass mode
Signed-off-by: lifubang <lifubang@acmcoder.com>
2020-03-12 15:53:47 +01:00
lifubang e1c29434b2 update linenoise to https://github.com/antirez/linenoise/tree/fc9667a81d43911a6690fb1e68c16e6e3bb8df05
Signed-off-by: lifubang <lifubang@acmcoder.com>
2020-03-12 15:53:47 +01:00
Jamie Scott e5a063bcb4 Remove default guidance in Redis.conf
Removing the default guidance in Redis.conf since this is not an available value.
2020-03-12 15:53:47 +01:00
Jamie Scott d28cbaf770 Update Redis.conf to improve TLS usability
When using TLS with a Redis.conf file the line for TLS reading tls-cert-file redis.crt tls-key-file redis.key is interpreted as one complete directive. I am separating this on two separate lines to improve usability so users do not get the below error.

ubuntu@ip-172-31-29-250:~/redis-6.0-rc1$ ./src/redis-server redis.conf 

*** FATAL CONFIG FILE ERROR ***
Reading the configuration file, at line 145
>>> 'tls-cert-file redis.crt tls-key-file redis.key'
wrong number of arguments
ubuntu@ip-172-31-29-250:~/redis-6.0-rc1$ vi redis.conf 
ubuntu@ip-172-31-29-250:~/redis-6.0-rc1$ ./src/redis-server redis.conf 
23085:C 04 Mar 2020 01:58:12.631 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
23085:C 04 Mar 2020 01:58:12.631 # Redis version=5.9.101, bits=64, commit=00000000, modified=0, pid=23085, just started
23085:C 04 Mar 2020 01:58:12.631 # Configuration loaded
23085:M 04 Mar 2020 01:58:12.632 * Increased maximum number of open files to 10032 (it was originally set to 1024).
2020-03-12 15:53:47 +01:00
Johannes Truschnigg 23d5e8b8e4 Signal systemd readiness atfer Partial Resync
"Partial Resynchronization" is a special variant of replication success
that we have to tell systemd about if it is managing redis-server via a
Type=Notify service unit.
2020-03-12 15:53:47 +01:00
Oran Agra 6173815408 fix for flaky psync2 test
*** [err]: PSYNC2: total sum of full synchronizations is exactly 4 in tests/integration/psync2.tcl
Expected 5 == 4 (context: type eval line 6 cmd {assert {$sum == 4}} proc ::test)

issue was that sometime the test got an unexpected full sync since it
tried to switch to the replica before it was in sync with it's master.
2020-03-12 15:53:47 +01:00
antirez 70e0e49952 ae.c: fix crash when resizing the event loop.
See #6964. The root cause is that the event loop may be resized from an
event callback itself, causing the event pointer to be invalid.
2020-03-12 15:53:47 +01:00
antirez b3e4aa6797 Fix release notes spelling mistake. 2020-03-06 11:09:52 +01:00
antirez e74e68c84f Redis 6 RC2. 2020-03-05 16:00:17 +01:00
qetu3790 4af0d7fd94 Fix not used constant in lru_test_mode.
LRU_CYCLE_PERIOD is defined,but not used.
2020-03-05 12:51:15 +01:00
hwware 6ef018785b add missing file marco 2020-03-05 12:51:15 +01:00
ShooterIT fe81d5c8a9 Avoid compiler warnings 2020-03-05 12:51:15 +01:00
antirez c2f01d7f9e RDB deletion: document it in example redis.conf. 2020-03-05 12:51:15 +01:00
antirez 127e09bca1 Make sync RDB deletion configurable. Default to no. 2020-03-05 12:51:15 +01:00
antirez a20303c623 Check that the file exists in removeRDBUsedToSyncReplicas(). 2020-03-05 12:51:15 +01:00
antirez baaf869fc3 Introduce bg_unlink(). 2020-03-05 12:51:14 +01:00
antirez be4bc1a5be Remove RDB files used for replication in persistence-less instances. 2020-03-05 12:51:14 +01:00
antirez 7a23b94559 Log RDB deletion in persistence-less instances. 2020-03-05 12:51:14 +01:00
antirez 07dc1b42fb Use a smaller getkeys global buffer.
The idea is that very few commands have a lot of keys, and when this
happens the allocation time becomes neglegible.
2020-03-05 12:51:14 +01:00
Oran Agra 10e71b3d01 Optimize temporary memory allocations for getKeysFromCommand mechanism
now that we may use it more often (ACL), these excessive calls to malloc
and free can become an overhead.
2020-03-05 12:51:14 +01:00
antirez edc0ed1415 Modules: reformat RM_Scan() top comment a bit. 2020-03-05 12:51:14 +01:00
antirez c5319612b4 Modules: more details in RM_Scan API top comment. 2020-03-05 12:51:14 +01:00
Oran Agra fff6b26ae3 RM_Scan disable dict rehashing
The callback approach we took is very efficient, the module can do any
filtering of keys without building any list and cloning strings, it can
also read data from the key's value. but if the user tries to re-open
the key, or any other key, this can cause dict re-hashing (dictFind does
that), and that's very bad to do from inside dictScan.

this commit protects the dict from doing any rehashing during scan, but
also warns the user not to attempt any writes or command calls from
within the callback, for fear of unexpected side effects and crashes.
2020-03-05 12:51:14 +01:00
Guy Benoish 650484604c Add RM_CreateStringFromDouble 2020-03-05 12:51:14 +01:00
Oran Agra 3144a278dd add no_auth to COMMAND INFO 2020-03-05 12:51:14 +01:00
Oran Agra afe0b16c02 module api docs for aux_save and aux_load 2020-03-05 12:51:14 +01:00
Guy Benoish df152b0ce7 streamReplyWithRangeFromConsumerPEL: Redundant streamDecodeID 2020-03-05 12:51:14 +01:00
antirez e3c1f43952 Show Redis version when not understanding a config directive.
This makes simpler to give people help when posting such kind of errors
in the mailing list or other help forums, because sometimes the
directive looks well spelled, but the version of Redis they are using is
not able to support it.
2020-03-05 12:51:14 +01:00
John Sully 5e63c0955d Fix CLANG build break 2020-03-04 17:24:17 -05:00
John Sully e523afa741 Fix CLANG build break 2020-03-04 17:24:00 -05:00
John Sully c2a98a60f9 Fix CLANG build break 2020-03-04 17:23:40 -05:00
John Sully 6bc92d6d31 Merge branch 'unstable' into redis_6_merge 2020-03-04 17:13:50 -05:00
John Sully 13cf8bbeb7 Merge branch 'unstable' into RELEASE_5 2020-03-04 17:13:32 -05:00
John Sully 60c0d4104d Add missing files from last checkin 2020-03-04 17:13:18 -05:00
John Sully 52fd4445a4 Merge branch 'unstable' into redis_6_merge 2020-03-04 17:13:01 -05:00
John Sully 693ba4ed4a Merge branch 'unstable' into RELEASE_5 2020-03-04 17:09:29 -05:00
John Sully 5397f0b033 Add extra logging when reporting errors from masters - especially in rreplay 2020-03-04 17:09:12 -05:00
John Sully 6c78ec7c41 MOTD refactor 2020-03-04 16:22:55 -05:00
John Sully fed4bd5269 min-clients-per-thread needs to use the new config code 2020-03-04 15:14:41 -05:00
John Sully c0882864f2 Update README.md 2020-03-03 13:58:48 -08:00
John Sully 1a7e2e663e Merge branch 'unstable' into redis_6_merge 2020-03-01 21:48:51 -05:00
John Sully faa87ed839 Merge branch 'unstable' into RELEASE_5 2020-03-01 21:43:21 -05:00
John Sully 83f585e30a Update tests to reflect new CRON name (keydb.cron) 2020-03-01 21:42:01 -05:00
benschermel 7a244bb847 Merge branch 'unstable' into RELEASE_5 2020-03-01 19:53:32 -05:00
John Sully 1b925119b5 Rename cron command to KEYDB.CRON and add help 2020-03-01 19:18:21 -05:00
John Sully 3eb847e28f Rename cron command to KEYDB.CRON and add help 2020-03-01 19:17:58 -05:00
John Sully 9dc8f30e47 Merge branch 'unstable' into redis_6_merge 2020-02-28 23:56:26 -05:00
John Sully 3ce6441a0c Bump version 2020-02-28 23:53:54 -05:00
John Sully 6d00a1a5cf Merge branch 'unstable' into RELEASE_5 2020-02-28 23:53:00 -05:00
John Sully aaca32a5cd Fix compile failure on raspberry pi machines, issue #141 2020-02-28 23:49:17 -05:00
John Sully 54c96d1995 Fix not respecting max clients config, issue #147 2020-02-28 22:04:42 -05:00
John Sully e5a2d59446 Fix issue #146 2020-02-28 21:21:05 -05:00
antirez 141c0679a0 Changelog: explain Redis 6 SPOP change. 2020-02-27 18:04:34 +01:00
Hengjian Tang 973297336f modify the read buf size according to the write buf size PROTO_IOBUF_LEN defined before 2020-02-27 18:02:30 +01:00
bodong.ybd fe902461f4 Fix spop return nil #4709 2020-02-27 18:02:30 +01:00
Ariel 15ea13245a fix ThreadSafeContext lock/unlock function names 2020-02-27 18:02:30 +01:00
antirez 9d4219ebac Fix SDS misuse in enumConfigSet(). Related to #6778. 2020-02-27 18:02:30 +01:00
Guy Benoish 4d12c37c54 XREADGROUP should propagate XCALIM/SETID in MULTI/EXEC
Use built-in alsoPropagate mechanism that wraps commands
in MULTI/EXEC before sending them to replica/AOF
2020-02-27 18:02:30 +01:00
antirez 8424306433 Remove useless comment from enumConfigSet(). 2020-02-27 18:02:30 +01:00
Oran Agra 12626ce9bb fix race in module api test for fork
in some cases we were trying to kill the fork before it got created
2020-02-27 18:02:30 +01:00
Ponnuvel Palaniyappan dafb94db79 Fix a potential overflow with strncpy 2020-02-27 18:02:30 +01:00
Guy Benoish 2ecab0b63a Modules: Do not auto-unblock clients if not blocked on keys 2020-02-27 18:02:30 +01:00
antirez ea697b6345 Improve aeDeleteEventLoop() top comment grammar. 2020-02-27 18:02:30 +01:00
Oran Agra 635321d47e fix github actions failing latency test for active defrag - part 2
it seems that running two clients at a time is ok too, resuces action
time from 20 minutes to 10. we'll use this for now, and if one day it
won't be enough we'll have to run just the sensitive tests one by one
separately from the others.

this commit also fixes an issue with the defrag test that appears to be
very rare.
2020-02-27 18:02:30 +01:00
wangyuan21 dd4798802c free time event when delete eventloop 2020-02-27 18:02:30 +01:00
Oran Agra 0b988fa9ec fix github actions failing latency test for active defrag
seems that github actions are slow, using just one client to reduce
false positives.

also adding verbose, testing only on latest ubuntu, and building on
older one.

when doing that, i can reduce the test threshold back to something saner
2020-02-27 18:02:30 +01:00
srzhao ecf3b2ef32 fix impl of aof-child whitelist SIGUSR1 feature. 2020-02-27 18:02:30 +01:00
Oran Agra 60096bc1a1 Fix latency sensitivity of new defrag test
I saw that the new defag test for list was failing in CI recently, so i
reduce it's threshold from 12 to 60.

besides that, i add / improve the latency test for that other two defrag
tests (add a sensitive latency and digest / save checks)

and fix bad usage of debug populate (can't overrides existing keys).
this was the original intention, which creates higher fragmentation.
2020-02-27 18:02:30 +01:00
meir@redislabs.com 2966132c52 Changed log level for module fork api from 'notice' to 'verbos'. 2020-02-27 18:02:30 +01:00
antirez b439542604 Tracking: optin/out implemented. 2020-02-27 18:02:30 +01:00
hwware 7277e5d8a8 format fix 2020-02-27 18:02:30 +01:00
antirez ef3551d149 Test engine: experimental change to avoid busy port problems. 2020-02-27 18:02:30 +01:00
hwware 1bb5ee9c68 fix potentical memory leaks 2020-02-27 18:02:30 +01:00
antirez 72c053519c Test engine: detect timeout when checking for Redis startup. 2020-02-27 18:02:30 +01:00
Oran Agra 349aa24511 Defrag big lists in portions to avoid latency and freeze
When active defrag kicks in and finds a big list, it will create a bookmark to
a node so that it is able to resume iteration from that node later.

The quicklist manages that bookmark, and updates it in case that node is deleted.

This will increase memory usage only on lists of over 1000 (see
active-defrag-max-scan-fields) quicklist nodes (1000 ziplists, not 1000 items)
by 16 bytes.

In 32 bit build, this change reduces the maximum effective config of
list-compress-depth and list-max-ziplist-size (from 32767 to 8191)
2020-02-27 18:00:47 +01:00
antirez 294c9af469 Test engine: better tracking of what workers are doing. 2020-02-27 18:00:47 +01:00
Guy Benoish b4ddc7b7ba XGROUP DESTROY should unblock XREADGROUP with -NOGROUP 2020-02-27 18:00:47 +01:00
hwware ba0270799e add missing subcommand description for debug oom 2020-02-27 18:00:47 +01:00
hayashier 73806f74ca fix typo from fss to rss 2020-02-27 18:00:47 +01:00
Guy Benoish 5d0890c0a8 Fix memory leak in test_ld_conv 2020-02-27 18:00:47 +01:00
antirez b6129f86c1 Test is more complex now, increase default timeout. 2020-02-27 18:00:47 +01:00
Madelyn Olson d1f22eaca4 Give an error message if you specify redirect twice 2020-02-27 18:00:47 +01:00
Madelyn Olson 762fbcb687 Minor CSC fixes and fixed documentation 2020-02-27 18:00:47 +01:00
antirez f15fb727a0 Tracking: fix max-keys configuration directive. 2020-02-27 18:00:46 +01:00
Itamar Haber e374573f30 Fixes segfault on calling trackingGetTotalKeys
... with CSC disabled
2020-02-27 18:00:46 +01:00
antirez 73d47d5749 Signal key as modified when expired on-access.
This fixes WATCH and client side caching with keys expiring because of
a synchronous access and not because of background expiring.
2020-02-27 18:00:46 +01:00
antirez b7cb28d501 Tracking: first set of tests for the feature. 2020-02-27 18:00:46 +01:00
antirez df83892760 Rax.c: populate data field after random walk. 2020-02-27 17:59:57 +01:00
antirez 1db7257107 Tracking: fix operators precedence error in bcast check. 2020-02-27 17:59:57 +01:00
antirez 0517da3618 Tracking: rename INFO field with total items. 2020-02-27 17:59:57 +01:00
antirez fe96e29d5f Tracking: fix behavior when switchinig from normal to BCAST. 2020-02-27 17:59:57 +01:00
antirez 3c16d6b32d Tracking: first conversion from hashing to key names. 2020-02-27 17:59:57 +01:00
antirez f21be1ec55 Tracking: fix sending messages bug + tracking off bug. 2020-02-27 17:59:57 +01:00
Oran Agra 3b4f147775 add no-slowlog option to RM_CreateCommand 2020-02-27 17:59:57 +01:00
antirez 6fb1aa2381 Tracking: BCAST: basic feature now works. 2020-02-27 17:59:57 +01:00
antirez d4fe79a174 Tracking: BCAST: broadcasting of keys in prefixes implemented. 2020-02-27 17:59:57 +01:00
antirez abb81c6351 Tracking: BCAST: registration in the prefix table. 2020-02-27 17:59:57 +01:00
antirez 77da960815 Tracking: BCAST: parsing of the options + skeleton. 2020-02-27 17:59:57 +01:00
antirez 3e8c69a9de Tracking: always reply with an array of keys. 2020-02-27 17:59:57 +01:00
antirez a788c373e6 Tracking: minor change of names and new INFO field. 2020-02-27 17:59:57 +01:00
John Sully d95bead383 Fix crash propogating stale keys 2020-02-26 21:49:40 -05:00
John Sully 51ea3cf30c Change Redis to KeyDB 2020-02-17 23:46:01 -05:00
John Sully 6ad6c1d780 Change Redis to KeyDB 2020-02-17 23:45:40 -05:00
John Sully 32f34f63fd Merge branch 'unstable' into redis_6_merge 2020-02-17 19:55:29 -05:00
John Sully 98aefac09b Add double unlock detection and improve fastlock_unlock assmebly 2020-02-17 19:54:05 -05:00
John Sully 5dc84850ad Incorrect use of std::atomic 2020-02-17 19:07:13 -05:00
John Sully 45b711f9d1 processEventsWhileBlocked not exception safe 2020-02-17 19:04:11 -05:00
John Sully 1ef187533c processEventsWhileBlocked not exception safe 2020-02-17 19:03:49 -05:00
John Sully 6a71de7835 Merge branch 'unstable' into redis_6_merge 2020-02-16 19:14:51 -05:00
John Sully 18496b6285 Support C++14 and remove dependency on future standard 2a 2020-02-16 19:04:56 -05:00
John Sully b9db899f6c Graceful shutdown of server threads when quit is requested 2020-02-16 18:53:46 -05:00
John Sully 4e9d2f08b1 Fix memory leak in RDB load 2020-02-16 18:42:14 -05:00
John Sully 06ad1c15d7 Fix memory leak in RDB load 2020-02-16 18:41:50 -05:00
John Sully 30a603ce05 Merge branch 'unstable' into redis_6_merge 2020-02-16 18:18:24 -05:00
John Sully 63ae679fe1 Merge branch 'unstable' of https://github.com/JohnSully/KeyDB into unstable 2020-02-16 18:12:11 -05:00
John Sully b92bbf4de8 Memory leak fix on config, and redisDb dtor 2020-02-16 17:08:00 -05:00
John Sully 4781eda722 Fix memory leak of ReplicaNestState on shutdown 2020-02-16 03:43:29 -05:00
John Sully 2fd93c5789 aeDeleteEventLoop use after free and leak fixes 2020-02-16 03:34:25 -05:00
Khem Raj 5e762d847c Mark extern definition of SDS_NOINIT in sds.h
This helps in avoiding multiple definition of this variable, its also
defined globally in sds.c

Signed-off-by: Khem Raj <raj.khem@gmail.com>
2020-02-12 14:17:54 +01:00
lifubang 54f5499aee correct help info for --user and --pass
Signed-off-by: lifubang <lifubang@acmcoder.com>
2020-02-12 14:17:54 +01:00
Seunghoon Woo 0c952b13d0 [FIX] revisit CVE-2015-8080 vulnerability 2020-02-12 14:17:54 +01:00
Guy Benoish dd34f70368 Diskless-load emptyDb-related fixes
1. Call emptyDb even in case of diskless-load: We want modules
   to get the same FLUSHDB event as disk-based replication.
2. Do not fire any module events when flushing the backups array.
3. Delete redundant call to signalFlushedDb (Called from emptyDb).
2020-02-12 14:17:54 +01:00
lifubang 5e042dbc05 fix ssl flag check for redis-cli
Signed-off-by: lifubang <lifubang@acmcoder.com>
2020-02-12 14:17:54 +01:00
Guy Benoish dcbe8bfad1 Exclude "keymiss" notification from NOTIFY_ALL
Because "keymiss" is "special" compared to the rest of
the notifications (Trying not to break existing apps
using the 'A' format for notifications)

Also updated redis.conf and module.c docs
2020-02-12 14:17:54 +01:00
Oran Agra 36caf2e42b update RM_SignalModifiedKey doc comment 2020-02-12 14:17:40 +01:00
Oran Agra 3067352a8d Add handling of short read of module id in rdb 2020-02-12 14:17:40 +01:00
Yossi Gottlieb 9baaf858f9 TLS: Update documentation. 2020-02-12 14:17:40 +01:00
Oran Agra 4440133e9a A few non-data commands that should be allowed while loading or stale
SELECT, and HELLO are commands that may be executed by the client
as soon as it connects, there's no reason to block them, preventing the
client from doing the rest of his sequence (which might just be INFO or
CONFIG, etc).

MONITOR, DEBUG, SLOWLOG, TIME, LASTSAVE are all non-data accessing
commands, which there's no reason to block.
2020-02-12 14:17:40 +01:00
Oran Agra c957794144 Memory leak when bind config is provided twice 2020-02-12 14:17:40 +01:00
Oran Agra 1333a46b7e fix maxmemory config warning
the warning condition was if usage > limit (saying it'll cause eviction
or oom), but in fact the eviction and oom depends on used minus slave
buffers.

other than fixing the condition, i add info about the current usage and
limit, which may be useful when looking at the log.
2020-02-12 14:17:40 +01:00
Oran Agra 8e7282eb3e Fix client flags to be int64 in module.c
currently there's no bug since the flags these functions handle are
always lower than 32bit, but still better fix the type to prevent future
bugs.
2020-02-12 14:17:40 +01:00
Oran Agra a678390e52 moduleRDBLoadError, add key name, and use panic rather than exit
using panic rather than exit means you get s stack trace of the code
path that experianced the error, and possibly other info.
2020-02-12 14:17:40 +01:00
Oran Agra f42ce57d0f stopAppendOnly resets aof_rewrite_scheduled
althouh in theory, users can do BGREWRITEAOF even if aof is disabled, i
suppose it is more common that the scheduled flag is set by either
startAppendOnly, of a failed initial AOFRW fork (AOF_WAIT_REWRITE)
2020-02-12 14:15:56 +01:00
Oran Agra 919fbf421d reduce repeated calls to use_diskless_load
this function possibly iterates on the module list
2020-02-12 14:15:56 +01:00
Oran Agra df096bc96b add SAVE subcommand to ACL HELP and top comment 2020-02-12 14:15:56 +01:00
Oran Agra 22e45d46fe freeClientAsync don't lock mutex if there's just one thread 2020-02-12 14:15:56 +01:00
Oran Agra a55e584706 DEBUG HELP - add PROTOCOL 2020-02-12 14:15:56 +01:00
Oran Agra ba28924418 move restartAOFAfterSYNC from replicaofCommand to replicationUnsetMaster
replicationUnsetMaster can be called from other places, not just
replicaofCOmmand, and all of these need to restart AOF
2020-02-12 14:15:56 +01:00
Guy Benoish 5a6cfbf4ca Some refactroing using getClientType instead of CLIENT_SLAVE 2020-02-12 14:15:56 +01:00
Guy Benoish fae306b374 Fix small bugs related to replica and monitor ambiguity
1. server.repl_no_slaves_since can be set when a MONITOR client disconnects
2. c->repl_ack_time can be set by a newline from a MONITOR client
3. Improved comments
2020-02-12 14:15:56 +01:00
Yossi Gottlieb 736309660f TLS: Some redis.conf clarifications. 2020-02-12 14:15:56 +01:00
Oran Agra 488e194787 config.c verbose error replies for CONFIG SET, like config file parsing
We noticed that the error replies for the generic mechanism for enums
are very verbose for config file parsing, but not for config set
command.

instead of replicating this code, i did a small refactoring to share
code between CONFIG SET and config file parsing.

and also renamed the enum group functions to be consistent with the
naming of other types.
2020-02-12 14:15:56 +01:00
Oran Agra c82ccf0670 memoryGetKeys helper function so that ACL can limit access to keys for MEMORY command 2020-02-12 14:15:56 +01:00
antirez 51c1a9f8fb ACL LOG: make max log entries configurable. 2020-02-12 14:15:35 +01:00
antirez ea1e1b12c9 ACL LOG: test for AUTH reason. 2020-02-12 14:15:35 +01:00
antirez 7379c78a9b ACL LOG: log failed auth attempts. 2020-02-12 14:15:35 +01:00
antirez 9f6e84f6be ACL LOG: implement a few basic tests. 2020-02-12 14:15:35 +01:00
antirez 82790e510f ACL LOG: also log ACL errors in the scripting/MULTI ctx. 2020-02-12 14:15:35 +01:00
antirez 943008ebac ACL LOG: implement LOG RESET. 2020-02-12 14:15:35 +01:00
antirez e271a61103 ACL LOG: group similar entries in a given time delta. 2020-02-12 14:15:35 +01:00
antirez f1974d5d67 ACL LOG: actually emit entries. 2020-02-12 14:15:35 +01:00
antirez d9b153c9f6 ACL LOG: implement ACL LOG subcommadn skeleton. 2020-02-12 14:15:35 +01:00
antirez 577fc4388b ACL LOG: data structures and initial functions. 2020-02-12 14:15:35 +01:00
John Sully 329519d4aa Merge branch 'unstable' into redis_6_merge 2020-02-11 03:48:08 -05:00
John Sully 52434a583a Fix race condition in allocating connections to threads 2020-02-11 03:44:28 -05:00
John Sully 18a5f46b61 Merge branch 'unstable' into redis_6_merge 2020-02-11 02:39:08 -05:00
John Sully 16a019dba0 Implement an error handler so bug #125 can't happen 2020-02-11 01:41:00 -05:00
John Sully 1a4c3224c9 Fix higher latency at low load by grouping clients to threads. This fixes slow perf in cluster benchmarks mentioned in issue #102 2020-02-11 01:00:21 -05:00
John Sully c3317686f8 Fix cases where duplicate RREPLAY is applied 2020-02-11 00:59:07 -05:00
John Sully 4622472123 Fix issue #119 2020-02-10 19:52:57 -05:00
John Sully f1748f8c76 Fix memory leak in cron 2020-02-10 18:26:04 -05:00
John Sully 0c101dccc8 Add missing test file 2020-02-10 18:15:29 -05:00
John Sully dc78bf1ccb CLANG build fix 2020-02-10 00:29:26 -05:00
John Sully a7cc3aac28 Ensure multi-master works for ring topologies 2020-02-10 00:25:03 -05:00
John Sully 339631530a Merge branch 'unstable' of https://github.com/JohnSully/KeyDB into unstable 2020-02-09 02:25:20 -05:00
John Sully fb2bdf7d05 Addmissing test file 2020-02-08 16:49:41 -05:00
John Sully c45f1328ac merge 5.0 unstable 2020-02-07 16:45:59 -05:00
John Sully 050d58007f Fix issue #137 and #132 2020-02-06 23:31:31 -05:00
John Sully 49d86746ed Add test to detect issue #137 and #132 2020-02-06 23:31:12 -05:00
Guy Benoish 6fe55c2f29 ld2string should fail if string contains \0 in the middle
This bug affected RM_StringToLongDouble and HINCRBYFLOAT.
I added tests for both cases.

Main changes:
1. Fixed string2ld to fail if string contains \0 in the middle
2. Use string2ld in getLongDoubleFromObject - No point of
   having duplicated code here

The two changes above broke RM_SaveLongDouble/RM_LoadLongDouble
because the long double string was saved with length+1 (An innocent
mistake, but it's actually a bug - The length passed to
RM_SaveLongDouble should not include the last \0).
2020-02-04 10:23:48 +01:00
Leo Murillo f7a94526dd Set ZSKIPLIST_MAXLEVEL to optimal value given 2^64 elements and p=0.25 2020-02-04 10:23:48 +01:00
antirez bbce3ba974 Add more info in the unblockClientFromModule() function. 2020-02-04 10:23:48 +01:00
WuYunlong eecfa9793e Fix lua related memory leak. 2020-02-04 10:23:48 +01:00
Guy Benoish 40295fb3fe Modules: Fix blocked-client-related memory leak
If a blocked module client times-out (or disconnects, unblocked
by CLIENT command, etc.) we need to call moduleUnblockClient
in order to free memory allocated by the module sub-system
and blocked-client private data

Other changes:
Made blockedonkeys.tcl tests a bit more aggressive in order
to smoke-out potential memory leaks
2020-02-04 10:23:48 +01:00
WuYunlong d2509811b7 Add tcl regression test in scripting.tcl to reproduce memory leak. 2020-02-04 10:23:48 +01:00
Yossi Gottlieb 29d4a1502a TLS: Fix missing initialization in redis-cli. 2020-02-04 10:23:48 +01:00
Oran Agra ec0c61da05 fix uninitialized info_cb var in module.c 2020-02-04 10:23:48 +01:00
John Sully ef4e11ecb8 module fixes 2020-02-04 01:22:25 -05:00
John Sully fd702e43f1 Fix cluster test failures from Redis 6 merge 2020-02-03 20:00:09 -05:00
John Sully 249e0fdd36 addRef memory model too restrictive 2020-02-01 21:52:17 -05:00
John Sully 6208118b13 RDB memory leaks 2020-01-30 17:57:10 -05:00
John Sully 73020b6a93 replication memory leaks 2020-01-30 17:56:24 -05:00
John Sully a9667e84ad Fix memory leak in cron 2020-01-30 17:55:48 -05:00
John Sully ab1524ff1a Merge branch 'redis_merge' into unstable 2020-01-30 16:05:40 -05:00
John Sully 1069d0b4bf CRON jobs were running half as often as required 2020-01-28 21:40:40 -05:00
John Sully 4a980f4ddb More threading fixes from merge 2020-01-28 17:54:00 -05:00
John Sully 9fbb9a551e reenable multithreading after merge 2020-01-27 19:59:04 -05:00
John Sully c24a7c03c2 cron reliability 2020-01-27 18:31:02 -05:00
John Sully 5a8280d577 fix expire test failures during overwrite 2020-01-27 18:26:23 -05:00
John Sully da83e84125 Fix most tests (still some failures) 2020-01-27 18:16:19 -05:00
John Sully ef9a3cadcf Merge remote-tracking branch 'redis/6.0' into redis_merge 2020-01-27 02:55:48 -05:00
John Sully c983d57f73 Missing cron files 2020-01-21 19:51:16 -05:00
John Sully 3204a39ada Initial implementation of the CRON command 2020-01-21 19:50:28 -05:00
John Sully f4319c1280 Bump version 2020-01-20 19:20:50 -05:00
antirez 721a39ddff Document I/O threads in redis.conf. 2020-01-17 10:47:38 +01:00
antirez cbabf779c2 Simplify #6379 changes. 2020-01-17 10:47:38 +01:00
antirez ecd17e819c Jump to right label on AOF parsing error.
Related to #6054.
2020-01-17 10:47:38 +01:00
antirez 8e9d19bc65 Change error message for #6775. 2020-01-17 10:47:38 +01:00
antirez 5be3a15a82 Setting N I/O threads should mean N-1 additional + 1 main thread. 2020-01-17 10:47:38 +01:00
WuYunlong 658749cc54 Free allocated sds in pfdebugCommand() to avoid memory leak. 2020-01-17 10:47:38 +01:00
antirez 1927932b43 Port PR #6110 to new connection object code. 2020-01-17 10:47:38 +01:00
Vasyl Melnychuk ba146d4c09 Make error when submitting command in incorrect context more explicit
So error message `ERR only (P)SUBSCRIBE / (P)UNSUBSCRIBE / PING / QUIT allowed in this context` will become
`ERR 'get' command submitted, but only (P)SUBSCRIBE / (P)UNSUBSCRIBE / PING / QUIT allowed in this context`
2020-01-17 10:47:38 +01:00
WuYunlong 47988c9666 Fix potential memory leak of clusterLoadConfig(). 2020-01-17 10:47:38 +01:00
antirez f2df5773b1 A few comments about main thread serving I/O as well.
Related to #6110.
2020-01-17 10:47:38 +01:00
WuYunlong cc90f79baf Fix potential memory leak of rioWriteBulkStreamID(). 2020-01-17 10:47:38 +01:00
zhaozhao.zz b3ff8a4b6d Threaded IO: use main thread to handle read work 2020-01-17 10:47:38 +01:00
zhaozhao.zz b1f2c51037 Threaded IO: use main thread to handle write work 2020-01-17 10:47:38 +01:00
John Sully c575e7df94 Avoid crash due to excessive posted functions for AOF rewrite 2020-01-11 16:34:30 -05:00
John Sully e603e57588 Merge branch 'unstable' into RELEASE_5 2020-01-10 17:46:23 -05:00
ShooterIT 7bbafc5639 Rename rdb asynchronously 2020-01-10 13:16:25 +01:00
Leo Murillo c7f752667e Fix bug on KEYS command where pattern starts with * followed by \x00 (null char). 2020-01-10 13:16:25 +01:00
Jamie Scott ed7ea13a2a Update to directive in redis.conf (missing s)
The directive tls-prefer-server-cipher is actually tls-prefer-server-ciphers in config.c. This results in a failed directive call shown below. This pull request adds the "s" in ciphers so that the directive is able to be properly called in config.c

ubuntu@ip-172-31-16-31:~/redis$ src/redis-server ./redis.conf 

*** FATAL CONFIG FILE ERROR ***
Reading the configuration file, at line 200
>>> 'tls-prefer-server-cipher yes'
Bad directive or wrong number of arguments
2020-01-10 13:16:25 +01:00
antirez 3be7762388 Free fakeclient argv on AOF error.
We exit later, so no bug fixed, but it is more correct.

See #6054, thanks to @ShooterIT for finding the issue.
2020-01-10 13:16:25 +01:00
antirez 15f6b74814 Git ignore: ignore more files. 2020-01-10 13:16:14 +01:00
Guy Benoish 1b5bf40c20 Blocking XREAD[GROUP] should always reply with valid data (or timeout)
This commit solves the following bug:
127.0.0.1:6379> XGROUP CREATE x grp $ MKSTREAM
OK
127.0.0.1:6379> XADD x 666 f v
"666-0"
127.0.0.1:6379> XREADGROUP GROUP grp Alice BLOCK 0 STREAMS x >
1) 1) "x"
   2) 1) 1) "666-0"
         2) 1) "f"
            2) "v"
127.0.0.1:6379> XADD x 667 f v
"667-0"
127.0.0.1:6379> XDEL x 667
(integer) 1
127.0.0.1:6379> XREADGROUP GROUP grp Alice BLOCK 0 STREAMS x >
1) 1) "x"
   2) (empty array)

The root cause is that we use s->last_id in streamCompareID
while we should use the last *valid* ID
2020-01-10 13:16:14 +01:00
John Sully 954c20edd9 Add support for incremental build with header files 2020-01-10 13:16:14 +01:00
WuYunlong 11c3afd711 Fix petential cluster link error.
Funcion adjustOpenFilesLimit() has an implicit parameter, which is server.maxclients.
This function aims to ajust maximum file descriptor number according to server.maxclients
by best effort, which is "bestlimit" could be lower than "maxfiles" but greater than "oldlimit".
When we try to increase "maxclients" using CONFIG SET command, we could increase maximum
file descriptor number to a bigger value without calling aeResizeSetSize the same time.
When later more and more clients connect to server, the allocated fd could be bigger and bigger,
and eventually exceeds events size of aeEventLoop.events. When new nodes joins the cluster,
new link is created, together with new fd, but when calling aeCreateFileEvent, we did not
check the return value. In this case, we have a non-null "link" but the associated fd is not
registered.

So when we dynamically set "maxclients" we could reach an inconsistency between maximum file
descriptor number of the process and server.maxclients. And later could cause cluster link and link
fd inconsistency.

While setting "maxclients" dynamically, we consider it as failed when resulting "maxclients" is not
the same as expected. We try to restore back the maximum file descriptor number when we failed to set
"maxclients" to the specified value, so that server.maxclients could act as a guard as before.
2020-01-10 13:16:14 +01:00
Yossi Gottlieb b752e83db5 Add REDISMODULE_CTX_FLAGS_MULTI_DIRTY. 2020-01-10 13:16:03 +01:00
hwware e16eb87465 typo fix in acl.c 2020-01-10 13:16:03 +01:00
Itamar Haber 35ea9d231b Adjusts 'io_threads_num' max to 128
Instead of 512, use the defined max from networking.c
2020-01-10 13:16:03 +01:00
antirez 3872912679 XCLAIM: Create the consumer only on successful claims.
Fixes #6744.
2020-01-08 12:31:39 +01:00
yz1509 b9a1530345 avoid sentinel changes promoted_slave to be its own replica. 2020-01-08 12:31:25 +01:00
John Sully d0654af74f Merge branch 'unstable' into RELEASE_5 2020-01-06 14:24:49 -05:00
John Sully 39acb312ef Bump version 2020-01-06 12:07:57 -05:00
John Sully b643c38204 Merge branch 'unstable' into RELEASE_5 2020-01-06 12:07:11 -05:00
antirez 5e7e5e6b61 Fix active expire division by zero.
Likely fix #6723.

This is what happens AFAIK: we enter the main loop where we expire stuff
until a given percentage of keys is still found to be logically expired.
There are however other potential exit conditions.

However the "sampled" variable is not always incremented inside the
loop, because we may found no valid slot as we scan the hash table, but
just NULLs ad dict entries. So when the do/while loop condition is
triggered at the end, we do (expired*100/sampled), dividing by zero if
we sampled 0 keys.
2020-01-01 18:10:39 +01:00
antirez e61dde8806 Fix duplicated CLIENT SETNAME reply.
Happened when we set the name to "" to cancel the name.
Was introduced during the RESP3 refactoring.

See #6036.
2019-12-29 15:46:31 +01:00
Guy Benoish cddf1da2e9 Stream: Handle streamID-related edge cases
This commit solves several edge cases that are related to
exhausting the streamID limits: We should correctly calculate
the succeeding streamID instead of blindly incrementing 'seq'
This affects both XREAD and XADD.

Other (unrelated) changes:
Reply with a better error message when trying to add an entry
to a stream that has exhausted last_id
2019-12-29 15:46:31 +01:00
Oran Agra 52ea44e53b config.c adjust config limits and mutable
- make lua-replicate-commands mutable (it never was, but i don't see why)
- make tcp-backlog immutable (fix a recent refactory mistake)
- increase the max limit of a few configs to match what they were before
the recent refactory
2019-12-29 15:46:31 +01:00
antirez 0f28ea16a7 Inline protocol: handle empty strings well.
This bug is from the first version of Redis. Probably the problem here
is that before we used an SDS split function that created empty strings
for additional spaces, like in "SET    foo          bar".
AFAIK later we replaced it with the curretn sdssplitarg() API that has
no such a problem. As a result, we introduced a bug, where it is no
longer possible to do something like:

    SET foo ""

Using the inline protocol. Now it is fixed.
2019-12-29 15:46:31 +01:00
antirez 00e5fefe6d Fix ip and missing mode in RM_GetClusterNodeInfo(). 2019-12-29 15:46:31 +01:00
antirez baafd30ba6 Redis 6.0 RC1. 2019-12-19 17:27:29 +01:00
Salvatore Sanfilippo d3a9dff6b9 Merge pull request #6615 from soloestoy/wrap-also-propagate-as-multi
Wrap also propagate as multi
2019-12-19 09:24:52 +01:00
Salvatore Sanfilippo 9a7b6a9f51 Merge pull request #5780 from soloestoy/lua-multi-more-clear
scripting: flag lua_client as CLIENT_MULTI after redis.replicate_command() immediately
2019-12-19 09:06:28 +01:00
Salvatore Sanfilippo f4b8197060 Merge pull request #6052 from jtru/better-systemd-integration-v2
Better systemd integration v2
2019-12-19 08:54:22 +01:00
antirez c5bc1c14c0 Modules: rewrite top function doc of AvoidReplicaTraffic(). 2019-12-18 17:11:03 +01:00
Salvatore Sanfilippo ef0b45ece8 Merge pull request #6497 from oranagra/avoid_replica_traffic
Add config and module API for AvoidReplicaTraffic
2019-12-18 17:06:05 +01:00
antirez 5a72c5058c Fix GEOHASH negative shifting in a more compatible way. 2019-12-18 12:57:32 +01:00
antirez e6e58e455c Revert "Geo: output 10 chars of geohash, not 11."
This reverts commit 009862ab7e.
2019-12-18 12:54:46 +01:00
antirez 5b9d3ac6c6 Avoid changing setKey() API after #6679 fix. 2019-12-18 11:58:02 +01:00
Salvatore Sanfilippo 9d1baa0705 Merge pull request #6679 from soloestoy/keepttl
Add a new SET option KEEPTTL and fix issue #5256
2019-12-18 11:52:08 +01:00
zhaozhao.zz 58554396d6 incrbyfloat: fix issue #5256 ttl lost after propagate 2019-12-18 15:44:51 +08:00
zhaozhao.zz 24044f3356 add a new SET option KEEPTTL that doesn't remove expire time 2019-12-18 15:20:36 +08:00
Salvatore Sanfilippo b7c78b7651 Merge pull request #5916 from madolson/dev-unstable-acl-module-pr
Add module APIs for custom authentication
2019-12-17 09:58:26 +01:00
antirez 4348d25fc4 Add some comment to PR #6144 changes. 2019-12-17 09:19:47 +01:00
Salvatore Sanfilippo f3d67b3842 Merge pull request #6144 from madolson/dev-unstable-cluster-down
Add configuration option for allowing reads on cluster down
2019-12-17 09:15:20 +01:00
Madelyn Olson 7f04a15311 Removed old auth context reference 2019-12-17 07:51:19 +00:00
Madelyn Olson a511a37bb7 Fixed some documentation 2019-12-17 07:49:21 +00:00
Madelyn Olson 7b3e3d6a13 Resolved merge miss 2019-12-16 23:40:19 -08:00
Madelyn Olson 12caffee61 Added a missed space in lua errors 2019-12-16 23:38:31 -08:00
Madelyn Olson 576a08908b Split error message so dependandent callers give a useful result 2019-12-16 23:34:37 -08:00
Madelyn Olson 44aa22c635 Added better exception handling around scripting and module 2019-12-16 23:33:53 -08:00
Madelyn Olson c95a582a74 Add configuration option for allowing reads on cluster down 2019-12-16 23:33:16 -08:00
Madelyn Olson f891b64a83 Tweaking the documentation 2019-12-17 07:28:55 +00:00
Madelyn Olson 67aa527b22 Added some documentation and fixed a test 2019-12-17 07:15:04 +00:00
Madelyn Olson 034dcf185c Add module APIs for custom authentication 2019-12-17 06:59:59 +00:00
Salvatore Sanfilippo e9b99c78df Merge pull request #6649 from chaten/unstable
Fix outdated information on max memory policies
2019-12-16 11:25:53 +01:00
antirez 901b7469b9 Clarify change in #6662. 2019-12-16 11:24:19 +01:00
antirez aa3f13761f Merge branch 'unstable' of github.com:/antirez/redis into unstable 2019-12-16 11:18:20 +01:00
Salvatore Sanfilippo 276a093753 Merge pull request #6662 from oranagra/fix_hz_div0
init server.hz early to avoid div by 0 during config file loading
2019-12-16 11:18:08 +01:00
antirez 096592506e Modules: allow to execute RM_Call() from the module init function. 2019-12-13 17:34:37 +01:00
Salvatore Sanfilippo ba28738083 Merge pull request #6663 from yossigo/fix-replace-value
Improve RM_ModuleTypeReplaceValue() API.
2019-12-12 18:00:07 +01:00
Yossi Gottlieb 0283db5883 Improve RM_ModuleTypeReplaceValue() API.
With the previous API, a NULL return value was ambiguous and could
represent either an old value of NULL or an error condition. The new API
returns a status code and allows the old value to be returned
by-reference.

This commit also includes test coverage based on
tests/modules/datatype.c which did not exist at the time of the original
commit.
2019-12-12 18:50:11 +02:00
Oran Agra ef6e263fde init server.hz early to avoid div by 0 during config file loading
since the refactory of config.c, it was initialized from config_hz in initServer
but apparently that's too late since the config file loading creates objects
which call LRU_CLOCK
2019-12-12 17:56:02 +02:00
Salvatore Sanfilippo 118db9eeae Merge pull request #6516 from IAmATeaPot418/patch-2
Add no-slowlog to acl command to prevent passwords
2019-12-12 09:41:19 +01:00
antirez d7a87d0ea4 redis-cli: some refactoring about not logging AUTH & co. 2019-12-12 09:40:36 +01:00
Salvatore Sanfilippo 81dc180323 Merge pull request #6524 from itamarhaber/acl-cli
Omits  `ACL SETUSER` from redis-cli's history
2019-12-12 09:33:48 +01:00
antirez 476333d192 Remove useless space from moduleLoad(). 2019-12-12 09:29:10 +01:00
Salvatore Sanfilippo c227f5e7bd Merge pull request #6257 from JohnSully/ModuleSecurity
Modules must have execute permissions to load
2019-12-12 09:27:47 +01:00
Salvatore Sanfilippo 2968d8e3cb Merge branch 'unstable' into ModuleSecurity 2019-12-12 09:27:14 +01:00
Oran Agra b5f3247ca5 Add module API for AvoidReplicaTraffic
This is useful to tell redis and modules to try to avoid doing things that may
increment the replication offset, and should be used when draining a master
and waiting for replicas to be in perfect sync before a failover.
2019-12-12 08:38:09 +02:00
Salvatore Sanfilippo c6fb9d0963 Merge pull request #6656 from oranagra/leak_rm_load_from_str
fix leak in RM_LoadDataTypeFromString, and save
2019-12-11 11:39:27 +01:00
Oran Agra a37cca3644 fix leak in RM_LoadDataTypeFromString, and save 2019-12-11 12:35:00 +02:00
antirez 25b36f8d65 Merge branch 'unstable' of github.com:/antirez/redis into unstable 2019-12-11 10:18:15 +01:00
antirez 441cd9713c Clients connected and bytes used is too spammy for verbose.
This message is there for ten years, but is hardly useful.
Moreover it is likely that it will fill an entire disk if log ratation
is not configured, for no good reasons.
2019-12-11 10:17:23 +01:00
antirez 29fddf3291 create-cluster script tailall option. 2019-12-11 10:16:25 +01:00
Michael Chaten 389faae58e Change description to be agnostic to the number of policies. 2019-12-09 11:27:53 -08:00
antirez e4b6070c0f Modules: more clarification about disconnection callback. 2019-12-09 12:15:38 +01:00
Salvatore Sanfilippo 14045adf92 Merge pull request #6653 from oranagra/client_tracking_flag
Add ULL suffix to CLIENT_TRACKING flag to prevent sign extension
2019-12-09 10:41:14 +01:00
Oran Agra 5941730c37 Add ULL suffix to CLIENT_TRACKING flag to prevent sign extension
the code in:
        c->flags &= ~(CLIENT_TRACKING|CLIENT_TRACKING_BROKEN_REDIR);
will do sign extension and turn on all the high 31 bits
no damage so far since we don't have any yet
2019-12-09 10:03:23 +02:00
Michael Chaten 52d83d57f9 Fix outdated information on max memory policies 2019-12-06 11:56:52 -08:00
Salvatore Sanfilippo 7e24e219fd Merge pull request #6645 from eliblight/fix-memcpy
Fix memcpy of id from key to be slightly safer.
2019-12-05 16:51:42 +01:00
Eran Liberty 08c3fe8063 - memcpy(&id,ri.key,ri.key_len);
+        memcpy(&id,ri.key,sizeof(id));

The memcpy from the key to the id reliease on the fact that this key
*should* be 8 bytes long as it was entered as such a few lines up the
code.

BUT if someone will change the code to the point this is no longer true,
current code can trash the stack which makes debugging very hard
while this fix will result in some garbage id, or even page fault.
Both are preferable to stack mangaling.
2019-12-05 13:37:11 +00:00
antirez 2271cf03f5 Modules: clarify when the disconnection callback is called. 2019-12-05 10:38:18 +01:00
antirez ecb742acc5 create-cluster script: allow additional options. 2019-12-03 17:52:04 +01:00
antirez 7b5f4b175b Modules: create timers in contexts without a client. 2019-12-03 16:21:23 +01:00
Salvatore Sanfilippo 89849c8b01 Merge pull request #6638 from oranagra/fix_module_short_read_test
fix crash in module short read test
2019-12-02 18:23:28 +01:00
Oran Agra 5661b19005 fix crash in module short read test 2019-12-02 19:17:35 +02:00
antirez 4092a75d85 Avoid collision with MacOS LIST_HEAD macro after #6384. 2019-12-02 09:13:29 +01:00
Salvatore Sanfilippo e5b5f9a2f6 Merge pull request #6384 from devnexen/apple_smaps_impl
Getting region date per process in Darwin
2019-12-02 09:02:08 +01:00
Salvatore Sanfilippo ce7ec725e3 Merge pull request #6624 from oranagra/config_c_step_3
Additional config.c refractory and bugfixes
2019-12-02 08:59:36 +01:00
Salvatore Sanfilippo ff14945e22 Merge pull request #5360 from youjh90/unstable
update leap year comment when div by 400
2019-12-02 08:56:08 +01:00
Salvatore Sanfilippo 731f10225b Merge pull request #6637 from andygua/add-missed-macro
Added the missed macro definition in slowlog.h
2019-12-02 08:51:05 +01:00
Salvatore Sanfilippo ddde700b64 Merge pull request #6634 from oranagra/fix_gen_write_load
revert an accidental test code change done as part of the tls project
2019-12-02 08:50:20 +01:00
Andy Guan 1ed5f84a79 Added the missed macro definition in slowlog.h 2019-12-02 00:28:41 -05:00
Oran Agra 07b365b7d7 revert an accidental test code change done as part of the tls project
it seems that commit b087dd1db6 accidentially changed
gen_write_load to not use deferred client, which causes them to be slower and not
generate high load which they should, making some tests less effecitive
2019-12-01 16:10:09 +02:00
Oran Agra 6b4366b85b config.c post refactory cleanup 2019-12-01 08:19:25 +02:00
antirez f5d48537f1 Fix Pi build needing -latomic. Issue #6275. 2019-11-29 17:35:59 +01:00
Oran Agra 18e72c5cc7 Converting more configs to use generic infra, and moving defaults to config.c
Changes in behavior:
- Change server.stream_node_max_entries from int64_t to long long, so that it can be used by the generic infra
- standard error reply instead of "repl-backlog-size must be 1 or greater" and such
- tls-port and a few TLS booleans were readable (config get) even when USE_OPENSSL was off (now they aren't)
- syslog-enabled, syslog-ident, cluster-enabled, appendfilename, and supervised didn't have a get (now they do)
- pidfile was initialized to NULL in InitServerConfig but had CONFIG_DEFAULT_PID_FILE in rewriteConfig (so the real default was "", but rewrite would cause it to be set), fixed the rewrite.
- TLS config in server.h was uninitialized (if no tls config args were provided)

Adding test for sanity and coverage
2019-11-28 11:24:57 +02:00
Oran Agra 28beb05aa3 More improvements and fixes to generic config infra
- Adding is_valid_fn and update_fn, both return 1 for success and 0 for failure with an optional error message.
- Bugfix in handling boundary check of unsigned numeric types (was boundaries as signed)
- Adding more numeric types to generic mechanism: uint, ulonglong, long, time_t, off_t
- More verbose error replies ("argument must be between" in out of range CONFIG SET (like config file parsing)
2019-11-28 11:11:07 +02:00
Oran Agra e0cc3c99d2 Additional config.c refactory and bugfixes
- add capability for each config to have a callback to check if value is valid and return error string
  will enable converting many of the remaining custom configs into generic ones (reducing the x4 repetition for set,get,config,rewrite)
- add capability for each config to  to run some update code after config is changed (only for CONFIG SET)
  will also enable converting many of the remaining custom configs into generic ones
- add capability to move default values from server.h and server.c to config.c
  will reduce many excess lines in server.h and server.c (plus, no need to rebuild the entire code base when a default change 8-))

other behavior changes:
- fix bug in bool config get (always returning 'yes')
- fix a bug in modifying jemalloc-bg-thread at runtime (didn't call set_jemalloc_bg_thread, due to bad merge conflict resolution (my fault))
- side effect when a failed attempt to enable activedefrag at runtime, we now respond with -ERR and not with -DISABLED
2019-11-26 16:52:28 +02:00
Salvatore Sanfilippo a1b654819c Merge pull request #6598 from oranagra/module-hook-test
try to fix an unstable test (module hook for loading progress)
2019-11-25 17:54:21 +01:00
Salvatore Sanfilippo a40669893e Merge pull request #6609 from daidaotong/macrofix
Add macro in bio.h
2019-11-25 10:04:28 +01:00
Salvatore Sanfilippo 6203ed0474 Merge pull request #6593 from daidaotong/movecommandfix
fix move command: WATCH/MULTI/EXEC mechanism breaks
2019-11-25 10:02:43 +01:00
Salvatore Sanfilippo d39afb687d Merge pull request #6563 from devnexen/fbsd_build_fix2
Add missing header
2019-11-25 09:45:29 +01:00
antirez ef5186d920 config.c: remove trailing spaces, trim comments to 80 cols. 2019-11-22 17:54:22 +01:00
Salvatore Sanfilippo a5e4c95cfd Merge pull request #6607 from madolson/dev-unstable-new-refactor
Refactored configs into types that can be generic
2019-11-22 17:42:42 +01:00
zhaozhao.zz 6b056d29f3 block: propagate BRPOPLPUSH as RPOPLPUSH when unblock 2019-11-22 16:38:49 +08:00
zhaozhao.zz 2c970532dc Propagation: flag module client as CLIENT_MULTI if needed
in case of nested MULTI/EXEC
2019-11-22 16:20:30 +08:00
zhaozhao.zz c73d70fb46 Propagation: propagate EXEC directly in lua script 2019-11-22 15:45:21 +08:00
zhaozhao.zz 37a10cef02 Propagation: wrap commands in also_propagate array with MULIT/EXEC
Random command like SPOP with count is replicated as
some SREM operations, and store them in also_propagate
array to propagate after the call, but this would break
atomicity.

To keep the command's atomicity, wrap also_propagate
array with MULTI/EXEC.
2019-11-22 15:42:49 +08:00
Madelyn Olson e8f4010e27 Changed a tab to a space 2019-11-22 06:44:12 +00:00
Madelyn Olson d50ae36e17 Fixed a bug with enum log printing 2019-11-22 06:28:50 +00:00
Madelyn Olson 818a4e0f70 Fixed some linux warnings 2019-11-22 05:49:52 +00:00
Madelyn Olson 519822bcf9 Added a flag for strings that are stored as NULL 2019-11-21 21:31:53 -08:00
zhaozhao.zz 73841e8c49 scripting: a little refactor about EVAL and redis.call()
Change server.lua_client's flag in a more explicit way.
2019-11-22 11:58:03 +08:00
zhaozhao.zz 097c4a6536 scripting: flag lua_client as CLIENT_MULTI after redis.replicate_commands() immediately
To avoid nested MULTI/EXEC, we check the lua_caller's flag,
if we are in the MULTI context we flag the lua_client as
CLIENT_MULTI, but it's not enough we shoud flag lua_client
as CLIENT_MULTI after redis.replicate_commands() immediately
or the first write command after redis.replicate_commands()
cannot know it's in an transaction, I know the missing CLIENT_MULTI
doesn't have any effect now, but it's a real bug and we should fix
it, in case someday we allow some dangerous command like BLPOP.
2019-11-22 11:51:18 +08:00
John Sully 7dd53182e0 Bump version 2019-11-21 20:16:42 -05:00
John Sully 53b95d830a Merge branch 'unstable' into RELEASE_5 2019-11-21 20:16:23 -05:00
Salvatore Sanfilippo b93945585a Merge pull request #6530 from oranagra/rm_get_server_info
Add module api for looking into INFO fields
2019-11-21 10:07:23 +01:00
Salvatore Sanfilippo 64c2508ee3 Merge branch 'unstable' into rm_get_server_info 2019-11-21 10:06:15 +01:00
Salvatore Sanfilippo f1f259de5b Merge pull request #6547 from guybe7/module_api_streams
Support streams in general module API functions
2019-11-21 10:03:50 +01:00
antirez da47d52c79 Recomment PR #6346. 2019-11-21 10:01:49 +01:00
Salvatore Sanfilippo fef1e63c70 Merge pull request #6346 from MeirShpilraien/expose_zmalloc_capabilities
Expose used memory to modules via redismodule api
2019-11-21 09:58:11 +01:00
Salvatore Sanfilippo 70469b76d6 Merge branch 'unstable' into expose_zmalloc_capabilities 2019-11-21 09:57:19 +01:00
Daniel Dai bfd8c1fdd2 macro fix
macro fix
2019-11-20 20:39:04 -05:00
Maxim Ivanov c7b68d10ea Prune leftover TODO comment
Is it sufficient... ? -- Yes it is. In standalone mode, we say READY=1
at the comment point; however in replicated mode, we delay sending
READY=1 until the replication sync completes.
2019-11-20 19:45:25 +02:00
Madelyn Olson 803d765d49 Refactored renaming types in config 2019-11-20 09:22:38 -08:00
Salvatore Sanfilippo c697edf4f4 Merge pull request #6559 from oranagra/active_defrag_tunables
Adjustments for active defrag defaults
2019-11-20 10:08:08 +01:00
Salvatore Sanfilippo 808394b77d Merge pull request #6603 from daidaotong/typofix
fix typo in scripting.acl
2019-11-20 10:06:33 +01:00
Daniel Dai 99b5696390 fix typo 2019-11-19 20:14:59 -05:00
Johannes Truschnigg 129d14e143 Auto-detect and link libsystemd at compile-time
This adds Makefile/build-system support for USE_SYSTEMD=(yes|no|*). This
variable's value determines whether or not libsystemd will be linked at
build-time.

If USE_SYSTEMD is set to "yes", make will use PKG_CONFIG to check for
libsystemd's presence, and fail the build early if it isn't
installed/detected properly.

If USE_SYSTEM is set to "no", libsystemd will *not* be linked, even if
support for it is available on the system redis is being built on.

For any other value that USE_SYSTEM might assume (e.g. "auto"),
PKG_CONFIG will try to determine libsystemd's presence, and set up the
build process to link against it, if it was indicated as being
installed/available.

This approach has a number of repercussions of its own, most importantly
the following: If you build redis on a system that actually has systemd
support, but no libsystemd-dev package(s) installed, you'll end up
*without* support for systemd notification/status reporting support in
redis-server. This changes established runtime behaviour.

I'm not sure if the build system and/or the server binary should
indicate this. I'm also wondering if not actually having
systemd-notify-support, but requesting it via the server's config,
should result in a fatal error now.
2019-11-19 18:55:44 +02:00
Johannes Truschnigg ec5681f0f1 Do not install SysV init-scripts on systemd-enabled hosts
Also, hint at example service unit files if systemd is detected. Thanks
to @mika for spotting a bug in the original iteration of this patch.
2019-11-19 18:55:44 +02:00
Johannes Truschnigg 5bbc112fb1 Provide example systemd service unit files for redis-server 2019-11-19 18:55:44 +02:00
Johannes Truschnigg 641c64ada1 Use libsystemd's sd_notify for communicating redis status to systemd
Instead of replicating a subset of libsystemd's sd_notify(3) internally,
use the dynamic library provided by systemd to communicate with the
service manager.

When systemd supervision was auto-detected or configured, communicate
the actual server status (i.e. "Loading dataset", "Waiting for
master<->replica sync") to systemd, instead of declaring readiness right
after initializing the server process.
2019-11-19 18:55:44 +02:00
Oran Agra ed2269762b try to fix an unstable test (module hook for loading progress)
there were two lssues, one is taht BGREWRITEAOF failed since the initial one was still in progress
the solution for this one is to enable appendonly from the server startup so there's no initial aofrw.

the other problem was 0 loading progress events, theory is that on some
platforms a sleep of 1 will cause a much greater delay due to the context
switch, but on other platform it doesn't. in theory a sleep of 100 micro
for 1k keys whould take 100ms, and with hz of 500 we should be gettering
50 events (one every 2ms). in practise it doesn't work like that, so trying
to find a sleep that would be long enough but still not cause the test to take
too long.
2019-11-19 15:01:51 +02:00
Salvatore Sanfilippo 45025713a8 Merge pull request #6541 from artix75/module-long-double
Module API: add support for long double conversions and replies
2019-11-19 12:15:53 +01:00
Salvatore Sanfilippo e7144fbed8 Merge branch 'unstable' into module-long-double 2019-11-19 12:15:45 +01:00
Salvatore Sanfilippo e916058f0b Merge pull request #6557 from oranagra/rm_lru_lfu_revized
rename RN_SetLRUOrLFU -> RM_SetLRU and RN_SetLFU
2019-11-19 11:58:07 +01:00
antirez fe5aea38c3 Simplify PR #6551 implementation. 2019-11-19 11:56:02 +01:00
Salvatore Sanfilippo 065872e001 Merge pull request #6551 from pvalsecc/sentinel_kill_pubsub
Redis sentinel kill pubsub client connections as well
2019-11-19 11:53:10 +01:00
antirez 446f24e66d Merge branch 'unstable' of github.com:/antirez/redis into unstable 2019-11-19 11:52:40 +01:00
Salvatore Sanfilippo b311a368e0 Merge pull request #6558 from oranagra/module_testrdb_leak
fix leak in module api rdb test
2019-11-19 11:49:43 +01:00
antirez 936e01e5bb Fix stream test after addition of 0-0 ID test. 2019-11-19 11:49:05 +01:00
Salvatore Sanfilippo 2151b82be9 Merge pull request #6571 from wangzhione/patch-2
Update adlist.h
2019-11-19 11:42:46 +01:00
Salvatore Sanfilippo 06a24b9cfa Merge pull request #6597 from yossigo/rm-call-extended-errors
Improve RM_Call() errno classification.
2019-11-19 11:41:32 +01:00
Salvatore Sanfilippo 8e95064408 Merge pull request #6134 from zhouyuan/fix_aof
aof: fix assignment for aof_fsync_offset
2019-11-19 11:40:34 +01:00
Salvatore Sanfilippo 397a8b57cc Merge pull request #6513 from oranagra/test_assertions
test infra: improve prints on failed assertions
2019-11-19 11:34:11 +01:00
antirez ce03d68332 Rename var to fixed_time_expire now that is more general. 2019-11-19 11:28:04 +01:00
antirez b42466b925 Fix patch provided in #6554. 2019-11-19 11:23:43 +01:00
Salvatore Sanfilippo 6a9851c7f8 Merge pull request #6554 from soloestoy/unblock-as-call
expires & blocking: handle ready keys as call()
2019-11-19 11:14:05 +01:00
Yossi Gottlieb 2d1e893b3e Improve RM_Call() errno classification.
RM_Call() will now use EBADF and ENONET in addition to EINVAL in order
to provide more information about errors (i.e. when return value is
NULL).
2019-11-19 12:10:48 +02:00
antirez 77ad9aaafe Merge branch 'unstable' of github.com:/antirez/redis into unstable 2019-11-19 11:10:35 +01:00
Salvatore Sanfilippo 2f41c8d518 Merge pull request #6512 from MeirShpilraien/scan_module_impl
Added scan implementation to module api.
2019-11-19 11:08:15 +01:00
Salvatore Sanfilippo 656e40eed2 Merge branch 'unstable' into scan_module_impl 2019-11-19 11:08:02 +01:00
antirez 5b80a41cad Remove additional space from comment. 2019-11-19 11:05:55 +01:00
Salvatore Sanfilippo 1530a3cfc6 Merge pull request #6577 from oranagra/module_doc
module docs, missing LOADING flag
2019-11-19 11:02:58 +01:00
Salvatore Sanfilippo 32a8301073 Merge pull request #4076 from yossigo/add_mt_replacevalue
Add RM_ModuleTypeReplaceValue.
2019-11-19 11:00:01 +01:00
Salvatore Sanfilippo 3d89210477 Merge pull request #3383 from yossigo/datatype_load_save
Redis Module API calls to allow re-use of data type RDB save/load.
2019-11-19 10:55:42 +01:00
Salvatore Sanfilippo a7122f4518 Merge pull request #6579 from oranagra/rm_reply_string_opt
Slightly more efficient RM_ReplyWithEmptyString
2019-11-19 10:54:00 +01:00
Salvatore Sanfilippo 70016f7861 Merge pull request #6574 from guybe7/fix_xadd_empty_id
XADD with ID 0-0 stores an empty key
2019-11-19 10:50:45 +01:00
Salvatore Sanfilippo 64a78f5ba3 Merge pull request #6596 from wangzhione/patch-5
Update mkreleasehdr.sh
2019-11-19 10:48:32 +01:00
喜欢兰花山丘 7c95e89ec3 Update mkreleasehdr.sh
fix date +%s errata
2019-11-19 17:23:47 +08:00
Daniel Dai 4ece8bd7a7 fix move command 2019-11-19 00:02:45 -05:00
antirez 3243252cb0 Expire cycle: document expire effort in redis.conf. 2019-11-18 18:11:38 +01:00
antirez e8ceba4e64 Expire cycle: set a buckets limit as well. 2019-11-18 17:47:19 +01:00
antirez 2ab51a644d Expire cycle: fix parameters computation. 2019-11-18 11:43:42 +01:00
antirez 1ccc270a2c Expire cycle: make expire effort configurable. 2019-11-18 11:33:44 +01:00
antirez 84b01f63db Expire cycle: introduce configurable effort. 2019-11-18 11:30:05 +01:00
antirez 2766805680 Expire cycle: tollerate less stale keys, expire cycle CPU in INFO. 2019-11-15 11:29:34 +01:00
antirez ffc7e509aa Expire cycle: scan hash table buckets directly. 2019-11-15 10:53:19 +01:00
antirez 2f6fe5ce3a Expire cycle: introduce the new state needed for the new algo. 2019-11-14 18:28:01 +01:00
antirez 8d50a8327e Rax library updated. 2019-11-14 12:48:54 +01:00
Oran Agra c426bbf3a5 Slightly more efficient RM_ReplyWithEmptyString
trimming talk about RESP protocol from API docs (should be independent to that anyway)
2019-11-14 09:46:46 +02:00
Oran Agra 2d30afc45f module docs, missing LOADING flag 2019-11-14 09:09:10 +02:00
Yossi Gottlieb 9c76875f41 Add RM_ModuleTypeReplaceValue.
This is a light-weight replace function, useful for use cases such as
realloc()ing an existing value, etc.  Using RM_ModuleTypeSetValue() in
such cases is wasteful and complex as it attempts to delete the old
value, call its destructor, etc.
2019-11-14 08:50:11 +02:00
Guy Benoish 4a12047c61 XADD with ID 0-0 stores an empty key
Calling XADD with 0-0 or 0 would result in creating an
empty key and storing it in the database.
Even worse, because XADD will reply with error the action
will not be replicated, creating a master-replica
inconsistency
2019-11-13 16:47:30 +05:30
喜欢兰花山丘 253d9d6d12 Update adlist.h
Update listGetFree keep format consistent
2019-11-13 10:14:45 +08:00
Oran Agra 0bc3dab095 Adjustments for active defrag defaults and tuning
Reduce default minimum effort, so that when fragmentation is just detected,
the impact on the latency will be minor.

Reduce the default maximum effort, mainly to prevent a case were a sudden
massive deletions, won't trigger an aggressive defrag that will cause latency.

When activedefrag is disabled mid-run, reset the 'running' info field, and
clear the scan cursor, so that when it'll be re-enabled, a new fresh scan will
start.

Clearing the 'running' variable is important since lowering the defragger
tunables mid-scan won't help, the defragger only considers new threshold when
a new scan starts, and during a scan it can only become more aggressive,
(when more severe fragmentation is detected), it'll never go less aggressive.
So by temporarily disabling activedefrag, one can lower th the tunables.

Removing the experimantal warning.
2019-11-12 15:05:29 +02:00
Oran Agra 0f8692b464 Add RM_ScanKey to scan hash, set, zset, changes to RM_Scan API
- Adding RM_ScanKey
- Adding tests for RM_ScanKey
- Refactoring RM_Scan API

Changes in RM_Scan
- cleanup in docs and coding convention
- Moving out of experimantal Api
- Adding ctx to scan callback
- Dont use cursor of -1 as an indication of done (can be a valid cursor)
- Set errno when returning 0 for various reasons
- Rename Cursor to ScanCursor
- Test filters key that are not strings, and opens a key if NULL
2019-11-11 16:05:55 +02:00
meir@redislabs.com 11c6ce812a Added scan implementation to module api.
The implementation expose the following new functions:
1. RedisModule_CursorCreate - allow to create a new cursor object for
keys scanning
2. RedisModule_CursorRestart - restart an existing cursor to restart the
scan
3. RedisModule_CursorDestroy - destroy an existing cursor
4. RedisModule_Scan - scan keys

The RedisModule_Scan function gets a cursor object, a callback and void*
(used as user private data).
The callback will be called for each key in the database proving the key
name and the value as RedisModuleKey.
2019-11-11 16:01:41 +02:00
David Carlier 960c25ac0a Add missing header 2019-11-11 13:06:08 +00:00
Oran Agra 28c20b4ef9 rename RN_SetLRUOrLFU -> RM_SetLRU and RN_SetLFU
- the API name was odd, separated to two apis one for LRU and one for LFU
- the LRU idle time was in 1 second resolution, which might be ok for RDB
  and RESTORE, but i think modules may need higher resolution
- adding tests for LFU and for handling maxmemory policy mismatch
2019-11-10 09:27:01 +02:00
Oran Agra 02f21113ab fix leak in module api rdb test
recently added more reads into that function, if a later read fails, i must
either free what's already allocated, or return the pointer so that the free
callback will release it.
2019-11-10 09:21:19 +02:00
zhaozhao.zz 7059eceeb0 expires & blocking: handle ready keys as call() 2019-11-08 19:06:51 +08:00
Patrick Valsecchi 9593ffde2e Redis sentinel kill pubsub client connections as well
When a redis instance becomes a slave, sentinel also kills pubsub
clients.

Closes #6545
2019-11-07 08:49:19 +01:00
antirez 0f026af185 Merge branch 'unstable' of github.com:/antirez/redis into unstable 2019-11-06 17:44:42 +01:00
meir@redislabs.com e45239e350 fix documentation 2019-11-06 12:26:03 +02:00
Guy Benoish 1833d008b3 Support streams in general module API functions
Fixes GitHub issue #6492
Added stream support in RM_KeyType and RM_ValueLength.
Also moduleDelKeyIfEmpty was updated, even though it has
no effect now (It will be relevant when stream type direct
API will be coded - i.e. RM_StreamAdd)
2019-11-06 15:52:19 +05:30
meir@redislabs.com c032dc45d1 changed GetUsedMemoryPresentage -> GetUsedMemoryRatio and return value between 0 and 1. 2019-11-06 12:17:09 +02:00
meir@redislabs.com aded138a59 return value between 0 to 100 instead of 0 to 1. 2019-11-06 11:28:52 +02:00
meir@redislabs.com 089ee5979d expose used memory via redismodule api
The exposed functions:
1. RedisModule_GetUsedMemoryPercentage - return the used memory
2. RedisModue_MallocSize - return for a given pointer, the amount of memory allocated for this pointer
2019-11-06 11:28:52 +02:00
antirez 8b2c0f9044 Update PR #6537: use a fresh time outside call().
One problem with the solution proposed so far in #6537 is that key
lookups outside a command execution via call(), still used a cached
time. The cached time needed to be refreshed in multiple places,
especially because of modules callbacks from timers, cluster bus, and
thread safe contexts, that may use RM_Open().

In order to avoid this problem, this commit introduces the ability to
detect if we are inside call(): this way we can use the reference fixed
time only when we are in the context of a command execution or Lua
script, but for the asynchronous lookups, we can still use mstime() to
get a fresh time reference.
2019-11-06 09:57:29 +01:00
antirez 824f5f0b7a Update PR #6537 patch to for generality.
After the thread in #6537 and thanks to the suggestions received, this
commit updates the original patch in order to:

1. Solve the problem of updating the time in multiple places by updating
it in call().
2. Avoid introducing a new field but use our cached time.

This required some minor refactoring to the function updating the time,
and the introduction of a new cached time in microseconds in order to
use less gettimeofday() calls.
2019-11-05 10:14:34 +01:00
zhaozhao.zz e542132b07 expires: refactoring judgment about whether a key is expired
Calling lookupKey*() many times to search a key in one command
may get different result.

That's because lookupKey*() calls expireIfNeeded(), and delete
the key when reach the expire time. So we can get an robj before
the expire time, but a NULL after the expire time.

The worst is that may lead to Redis crash, for example
`RPOPLPUSH foo foo` the first time we get a list form `foo` and
hold the pointer, but when we get `foo` again it's expired and
deleted. Now we hold a freed memory, when execute rpoplpushHandlePush()
redis crash.

To fix it, we can refactor the judgment about whether a key is expired,
using the same basetime `server.cmd_start_mstime` instead of calling
mstime() everytime.
2019-11-05 09:56:39 +01:00
Yossi Gottlieb 5350e7669e Add ModuleDataType to/from string serialization.
Add two new functions that leverage the RedisModuleDataType mechanism
for RDB serialization/deserialization and make it possible to use it
to/from arbitrary strings:

* RM_SaveDataTypeToString()
* RM_LoadDataTypeFromString()
2019-11-04 21:53:59 +02:00
Salvatore Sanfilippo a15a5d7097 Merge pull request #6540 from oranagra/simple_module_api_tests
Test coverage for new module APIs: dbsize, flushall, randomkey, lru get/set
2019-11-04 18:52:31 +01:00
artix a68c19df6c RM_CreateStringFromLongDouble: use new ld2string 'mode' type 2019-11-04 18:49:56 +01:00
artix 060af1858d Fix RedisModule_ReplyWithLongDouble ptr definition, add tests 2019-11-04 18:49:50 +01:00
artix 60ec2b78b3 Start support for long double in modules
New API:

- RedisModule_StringToLongDouble
- RedisModule_CreateStringFromLongDouble
- RedisModule_ReplyWithLongDouble
2019-11-04 18:49:42 +01:00
Oran Agra 3adf10b809 Test coverage for new module APIs: dbsize, flushall, randomkey, lru get/set 2019-11-04 19:30:31 +02:00
Salvatore Sanfilippo 576a7b8caf Merge pull request #6539 from kristoff-it/unstable
fix unreported overflow in autogerenared stream IDs
2019-11-04 17:37:28 +01:00
Salvatore Sanfilippo bbdbfabf9e Merge pull request #6531 from oranagra/rm_save_long_double
Module API for loading and saving long double
2019-11-04 17:33:35 +01:00
Loris Cro b12d2f65d6 fix unreported overflow in autogerenared stream IDs 2019-11-04 16:36:06 +01:00
Loris Cro 3f14bfd809 Merge pull request #1 from antirez/unstable
update to latest unstable
2019-11-04 16:12:09 +01:00
Salvatore Sanfilippo 720d1fd3bb Merge pull request #6486 from oranagra/module_lru_lfu
Module API for controlling LRU and LFU, and OpenKey without TOUCH
2019-11-04 11:05:08 +01:00
Salvatore Sanfilippo 18b62a51e2 Merge branch 'unstable' into module_lru_lfu 2019-11-04 11:04:49 +01:00
Salvatore Sanfilippo 327cb45ed5 Merge pull request #6532 from oranagra/rm_misc_commands
Module API for PUBLISH, FLUSHALL, RANDOMKEY, DBSIZE
2019-11-04 11:02:37 +01:00
Salvatore Sanfilippo 23c681b51d Merge pull request #6535 from guybe7/module_block_on_keys_tests
Modules: Test RedisModule_BlockClientOnKeys
2019-11-04 10:53:59 +01:00
Salvatore Sanfilippo dcc4347c53 Merge pull request #6514 from oranagra/module_hooks
Modules hooks: complete missing hooks for the initial set of hooks
2019-11-04 10:37:16 +01:00
Salvatore Sanfilippo da44c54897 Merge branch 'unstable' into module_hooks 2019-11-04 10:37:06 +01:00
Guy Benoish b81f486c2f Modules: Test RedisModule_BlockClientOnKeys 2019-11-04 14:20:24 +05:30
Oran Agra 0423309768 Add RM_ServerInfoGetFieldUnsigned
rename RM_ServerInfoGetFieldNumerical RM_ServerInfoGetFieldSigned
move string2ull to util.c
fix leak in RM_GetServerInfo when duplicate info fields exist
2019-11-04 08:50:29 +02:00
Oran Agra deebed23e1 Adding RM_ServerInfoGetFieldC 2019-11-04 07:57:52 +02:00
Oran Agra 87332ce524 Module API for PUBLISH, FLUSHALL, RANDOMKEY, DBSIZE 2019-11-04 07:40:52 +02:00
Oran Agra 779aebc91c Module API for loading and saving long double
looks like each platform implements long double differently (different bit count)
so we can't save them as binary, and we also want to avoid creating a new RDB
format version, so we save these are hex strings using "%La".

This commit includes a change in the arguments of ld2string to support this.
as well as tests for coverage and short reads.

coded by @guybe7
2019-11-03 16:42:31 +02:00
Oran Agra 4d580438b0 Add module api for looking into INFO fields
- Add RM_GetServerInfo and friends
- Add auto memory for new opaque struct
- Add tests for new APIs

other minor fixes:
- add const in various char pointers
- requested_section in modulesCollectInfo was actually not sds but char*
- extract new string2d out of getDoubleFromObject for code reuse

Add module API for
2019-11-03 15:02:25 +02:00
antirez fdaea2a7a7 Modules: fix thread safe context creation crash.
See #6525, this likely creates a NULL deference if the client was
terminated by Redis between the creation of the blocked client and the
creation of the thread safe context.
2019-10-31 18:07:33 +01:00
antirez 1ee195d255 Merge branch 'unstable' of github.com:/antirez/redis into unstable 2019-10-31 17:55:18 +01:00
antirez dd5feec5e8 Modules: block on keys: fix stale comment. 2019-10-31 17:45:07 +01:00
antirez 825adcf3f5 Modules: block on keys: finish implementing RM_UnblockClient(). 2019-10-31 17:39:05 +01:00
Itamar Haber 9f32d0eed2 Ommits from redis-cli's history 2019-10-31 15:45:52 +02:00
antirez 37bf3e18cb Modules: block ok keys: improve example. 2019-10-31 12:31:22 +01:00
antirez 66f55bc5c1 Modules: block on keys: fix bugs in processing order. 2019-10-31 12:23:55 +01:00
antirez 629081f839 Modules: block on keys: fix the top comments. 2019-10-31 11:43:45 +01:00
antirez 91f4bdc9f9 Modules: block on keys: use a better interface.
Using the is_key_ready() callback plus the reply callback later, creates
different issues AFAIK:

1. More complex API.
2. We need to call the reply callback() ASAP if the is_key_ready()
interface returned success, however the internals do not work in that
way, so when the reply callback is called the setup could be different.
To fix that, there is to break the current design that handles the
unblocked clients asyncrhonously, and run the list ASAP.
2019-10-31 11:35:07 +01:00
antirez 4534960b29 Modules: remove spurious call from moduleHandleBlockedClients().
Now we handle propagation when we free the context.
2019-10-31 10:32:59 +01:00
antirez 228bc89ecb Modules: block on keys: example on hellotype.c. 2019-10-31 10:30:54 +01:00
antirez 215b72c0ba Modules: block on keys: implement the internals. 2019-10-30 10:57:44 +01:00
antirez fb6110ac20 Modules: block on keys: export APIs. 2019-10-30 10:20:28 +01:00
antirez 3649568ff2 Modules: block on keys functions layout and mechanism. 2019-10-30 10:11:58 +01:00
Jamie Scott 5eb2290f68 Add no-slowlog to acl command to prevent passwords
Adding no-slowlog to acl command to prevent acl passwords from showing in slowlog
2019-10-29 15:10:07 -07:00
Oran Agra 51c3ff8d75 Modules hooks: complete missing hooks for the initial set of hooks
* replication hooks: role change, master link status, replica online/offline
* persistence hooks: saving, loading, loading progress
* misc hooks: cron loop, shutdown, module loaded/unloaded
* change the way hooks test work, and add tests for all of the above

startLoading() now gets flag indicating what is loaded.
stopLoading() now gets an indication of success or failure.
adding startSaving() and stopSaving() with similar args and role.
2019-10-29 17:59:09 +02:00
Oran Agra a0cfd519e3 test infra: improve prints on failed assertions
sometimes we have several assertions with the same condition in the same test
at different stages, and when these fail (the ones that print the condition
text) you don't know which one it was. other assertions didn't print the
condition text (variable names), just the expected and unexpected values.

So now, all assertions print context line, and conditin text.

besides, one of the major differences between 'assert' and 'assert_equal',
is that the later is able to print the value that doesn't match the expected.
if there is a rare non-reproducible failure, it is helpful to know what was
the value the test encountered and how far it was from the threshold.

So now, adding assert_lessthan and assert_range that can be used in some places.
were we used just 'assert { a > b }' so far.
2019-10-29 17:38:12 +02:00
Oran Agra e978bdf9ef Module API for controlling LRU and LFU, and OpenKey without TOUCH
Some commands would want to open a key without touching it's LRU/LFU
similarly to the OBJECT or DEBUG command do.

Other commands may want to implement logic similar to what RESTORE
does (and in the future MIGRATE) and get/set the LRU or LFU.
2019-10-29 15:03:05 +02:00
Salvatore Sanfilippo 764b420f5f Merge pull request #6487 from oranagra/module_modified_key
Module API for explicit SignalModifiedKey instead of implicit one.
2019-10-29 10:40:22 +01:00
Salvatore Sanfilippo 568dfd4b49 Merge pull request #6470 from gkorland/CI
Add Github action for CI
2019-10-29 10:25:34 +01:00
Salvatore Sanfilippo 94973fd0eb Merge pull request #6508 from guybe7/fix_module_notify_comp_error
Fix compilation error introduced by cee6dead3
2019-10-29 10:07:45 +01:00
Salvatore Sanfilippo 90169cdf16 Merge pull request #6495 from oranagra/rm_call_argv
Module api RM_CallArgv and test
2019-10-29 10:07:05 +01:00
Guy Benoish 43e2a398f9 Fix compilation error introduced by cee6dead3
Need to add calls to REDISMODULE_API_FUNC...
2019-10-28 17:58:07 +05:30
Salvatore Sanfilippo 469a6e5f1a Merge pull request #6481 from guybe7/modules_notify
Modules: Allow notifying custom keyspace events
2019-10-28 12:16:04 +01:00
Oran Agra 0399b5a27e Module api tests for RM_Call
Adding a test for coverage for RM_Call in a new "misc" unit
to be used for various short simple tests

also solves compilation warnings in redismodule.h and fork.c
2019-10-28 12:39:57 +02:00
Guy Benoish cee6dead3c Modules: Allow notifying custom keyspace events
Also, add an API for getting server.notify_keyspace_events

Other (unrelated) changes:
Add RM_GetKeynameFromModuleKey
2019-10-28 15:53:35 +05:30
antirez cfcb475435 LOLWUT 6: always pick a default color sequence. 2019-10-28 11:10:58 +01:00
Oran Agra a12f07792f Module API for explicit SignalModifiedKey instead of implicit one.
This commit also fixes an uninitialized module struct member (that luckily never got released)
2019-10-28 12:09:25 +02:00
antirez 8337e083b6 Merge branch 'unstable' of github.com:/antirez/redis into unstable 2019-10-28 11:09:09 +01:00
Salvatore Sanfilippo c328c807e7 Merge pull request #6496 from oranagra/module_tests_valgrind
Make module tests pass with valgrind, and fix a leak in diskless load
2019-10-28 10:05:52 +01:00
Salvatore Sanfilippo d2658f8387 Merge pull request #6493 from yossigo/modules-make-cleanup
Modules: improve tests Makefile.
2019-10-28 10:00:40 +01:00
Salvatore Sanfilippo 5aab012676 Merge pull request #6498 from oranagra/rm_latency_add_sample
Module API for LatencyAddSample
2019-10-28 09:59:26 +01:00
Salvatore Sanfilippo e097c99681 Merge pull request #4761 from WJWH/minor-typos
Fix some minor typos in comments
2019-10-28 09:58:38 +01:00
Salvatore Sanfilippo 6f2fc34228 Merge pull request #6502 from happynote3966/typo-fix
fix comment typo in redis-cli.c
2019-10-28 09:57:43 +01:00
Salvatore Sanfilippo c6c65d9ab3 Merge pull request #6503 from cstrotm/unstable
Typo fix: kill -> still
2019-10-28 09:57:19 +01:00
Salvatore Sanfilippo d070221707 Merge pull request #6505 from swilly22/replaywith-module-API-additions
Introduce ReplyWithVerbatimString, ReplyWithEmptyArray, ReplyWithNull…
2019-10-28 09:56:56 +01:00
Salvatore Sanfilippo 24fdc87fe0 Merge pull request #4994 from soloestoy/module-blocked-client
Modules: make unloading module more safe
2019-10-28 09:55:11 +01:00
swilly22 56a7c45521 Introduce ReplyWithVerbatimString, ReplyWithEmptyArray, ReplyWithNullArray and ReplyWithEmptyString to redis module API 2019-10-28 08:50:25 +02:00
zhaozhao.zz c74398e130 Modules: make unloading module more safe
As we know if a module exports module-side data types,
unload it is not allowed. This rule is the same with
blocked clients in module, because we use background
threads to implement module blocked clients, and it's
not safe to unload a module if there are background
threads running. So it's necessary to check if any
blocked clients running in this module when unload it.

Moreover, after that we can ensure that if no modules,
then no module blocked clients even module unloaded.
So, we can call moduleHandleBlockedClients only when
we have installed modules.
2019-10-28 11:31:58 +08:00
Carsten Strotmann cb2ca37f88 Typo fix: kill -> still 2019-10-25 19:38:45 +02:00
happynote3966 cd02c5944f fix comment typo in redis-cli.c 2019-10-25 22:31:55 +09:00
Wander Hillen fb1f4f4e7e Merge branch 'unstable' into minor-typos 2019-10-25 10:18:26 +02:00
John Sully a91f58b8fd Fix potential race in pubsub 2019-10-25 02:44:14 -04:00
John Sully 3013799664 bump version 2019-10-25 01:00:28 -04:00
John Sully f8220a658e Merge branch 'unstable' into RELEASE_5 2019-10-24 23:39:18 -04:00
Oran Agra e3b1d6d3ad Module API for LatencyAddSample 2019-10-24 14:24:55 +03:00
Yossi Gottlieb 183d264e9e Modules: improve tests Makefile.
Removes some boilerplate per module, add clean target.
2019-10-24 13:21:35 +03:00
antirez 5dd1130272 Remove trailing space from server.c. 2019-10-24 11:20:15 +02:00
antirez 6e98214f74 Modules hooks: test flush event. 2019-10-24 10:51:03 +02:00
antirez 39f2ab595c Modules hooks: initial Tcl test file. 2019-10-24 10:41:25 +02:00
Oran Agra be352633a0 Make module tests pass with valgrind, and fix a leak in diskless load 2019-10-24 09:45:25 +03:00
antirez 7cefbeaad5 Modules hooks: select DB zero by default. 2019-10-23 18:51:44 +02:00
antirez d3266d516b Modules hooks: test module draft. 2019-10-23 18:42:02 +02:00
antirez dddfde85ef Modules hooks: fix memory leak in example module. 2019-10-23 18:39:53 +02:00
antirez 61ac2aef2b Modules hooks: for nested calls, create new fake clients.
We can't use the same client at the same time when re-entering the hook.
2019-10-23 18:39:53 +02:00
antirez 8cccc6b468 Modules hooks: don't call the hook for fake clients. 2019-10-23 18:39:53 +02:00
antirez 83de18805c Modules hooks: do not re-enter in hooks.
Calling a module hook callback may result in callback operations in turn
triggering other events the module is subscribed too. We don't want to
trigger those, it's unsafe and quite confusing, and to do it correcly we
would need to maintain an event list: quite a more complex
implementation.
2019-10-23 18:39:53 +02:00
antirez 89af7e4c85 Modules hooks: select the right DB for FLUSHDB events. 2019-10-23 18:39:53 +02:00
antirez bc1ef48e56 Modules hooks: FLUSHDB event example. 2019-10-23 18:39:53 +02:00
antirez ed833c9f99 Modules hooks: implement the FLUSHDB event. 2019-10-23 18:39:53 +02:00
antirez 8ec2fc3969 Modules hooks: unify structures definitions. 2019-10-23 18:39:53 +02:00
antirez db8c9a8520 Modules hooks: document what yet to implement WIP 2. 2019-10-23 18:39:53 +02:00
antirez 18d1607f5e Modules hooks: document what yet to implement WIP 1. 2019-10-23 18:39:53 +02:00
antirez c379fdbae2 Modules hooks: do more in example client callback. 2019-10-23 18:39:53 +02:00
antirez cf52e0ad44 Module hooks: fixes and an example module. 2019-10-23 18:39:53 +02:00
antirez 94c50314f1 Modules hooks: fix sub event in calls from client change. 2019-10-23 18:39:53 +02:00
antirez 019ac37e9c Modules hooks: fix define / linker issues. Implement one test event. 2019-10-23 18:39:53 +02:00
antirez d54652682d Modules hooks: handle module unloading + API export. 2019-10-23 18:39:53 +02:00
antirez 1e78681df8 Modules hooks: fix a leak and a few more issues. 2019-10-23 18:39:53 +02:00
antirez b9af7e24e4 Modules hooks: event firing logic. 2019-10-23 18:39:53 +02:00
antirez 6e56f513b4 Modules: refactoring of RM_GetClientInfoById(). 2019-10-23 18:39:53 +02:00
antirez 21d8544128 Modules: fix RedisModule_GetClientInfoById() example. 2019-10-23 18:39:53 +02:00
antirez e9b5484ddd Modules: export GetClientInfoById(). 2019-10-23 18:39:53 +02:00
antirez 6ea8641ae0 Modules: add the client ID to the client info structure. 2019-10-23 18:39:53 +02:00
antirez d2d74ca0f3 Modules: RM_GetClientInfoById() draft. 2019-10-23 18:39:53 +02:00
antirez bebb2e1994 Modules hooks: a first version of events and some API. 2019-10-23 18:39:53 +02:00
antirez 3170f63330 Modules hooks: initial design of data structures. 2019-10-23 18:39:53 +02:00
antirez ed2ba31e4c Modules: fix moduleCreateArgvFromUserFormat() casting bug.
In 32 bit systems casting to "long" will cut the result to 32 bit.
2019-10-23 18:39:14 +02:00
John Sully 3fac516950 Disable multithreaded KEYS due to bugs 2019-10-21 12:21:46 -04:00
Guy Korland 0381f14acc add CI action 2019-10-20 10:04:25 +03:00
Guy Korland 1ac30300f0 Merge branch 'unstable' of github.com:antirez/redis into unstable 2019-10-20 09:59:23 +03:00
antirez 673c9d7029 module: fix propagation API bug. 2019-10-17 18:28:34 +02:00
antirez 97f6e314c1 Modules: allow to check for AOF loading client. 2019-10-17 18:10:50 +02:00
antirez 8651e6a10d Modules: add new flags to context, replica state + more. 2019-10-17 17:49:33 +02:00
John Sully 2f65aa2337 Merge branch 'unstable' into RELEASE_5 2019-10-16 12:47:06 -04:00
Salvatore Sanfilippo efb6495a44 Merge pull request #6236 from yossigo/poc/conns
Abstract Connections I/O API & TLS Support
2019-10-16 17:42:35 +02:00
Yossi Gottlieb 8e29b0b22b Fix Makefile merge issue. 2019-10-16 17:31:02 +03:00
Yossi Gottlieb 0db3b0a0ff Merge remote-tracking branch 'upstream/unstable' into tls 2019-10-16 17:08:07 +03:00
Salvatore Sanfilippo b8e02f2b40 Merge pull request #5926 from JimB123/unstable
Addition of RedisModule_OnUnload hook
2019-10-16 11:13:12 +02:00
Yossi Gottlieb c469f6ad9e Code review minor changes (names, comments). 2019-10-15 17:21:51 +03:00
Yossi Gottlieb 71f10de4de Conns: write() 0 retval should not trigger error. 2019-10-15 17:20:58 +03:00
Yossi Gottlieb 0132189007 Fix compile warnings when BUILD_TLS=no. 2019-10-15 15:24:32 +03:00
antirez dd29d44136 Test: fix implementation-dependent test after code change. 2019-10-10 15:22:42 +02:00
Salvatore Sanfilippo 8e3d910d4e Merge pull request #6429 from charsyam/feature/typo-slave
[trivial] fix typos salves to slaves in replication.c
2019-10-10 14:56:43 +02:00
Salvatore Sanfilippo 363604a27d Merge pull request #6437 from OMG-By/redis-omg_by
there should is AUTH && HELLO non authenticated state.
2019-10-10 14:55:03 +02:00
Salvatore Sanfilippo 1677f4223c Merge pull request #6443 from guybe7/fix_stream_rax_node_limits
Fix usage of server.stream_node_max_*
2019-10-10 14:49:35 +02:00
Salvatore Sanfilippo 14a9da0613 Merge pull request #6145 from oranagra/jemalloc_purge_bg
purge jemalloc after flush, and enable background purging thread
2019-10-10 14:48:59 +02:00
Salvatore Sanfilippo 1d9d27c91a Merge pull request #6438 from daidaotong/unstable
resolve typo in test module
2019-10-10 11:18:47 +02:00
antirez 509fe9cba1 Merge branch 'unstable' of github.com:/antirez/redis into unstable 2019-10-10 10:31:43 +02:00
antirez 747be463d2 Cluster: fix memory leak of cached master.
This is what happened:

1. Instance starts, is a slave in the cluster configuration, but
actually server.masterhost is not set, so technically the instance
is acting like a master.

2. loadDataFromDisk() calls replicationCacheMasterUsingMyself() even if
the instance is a master, in the case it is logically a slave and the
cluster is enabled. So now we have a cached master even if the instance
is practically configured as a master (from the POV of
server.masterhost value and so forth).

3. clusterCron() sees that the instance requires to replicate from its
master, because logically it is a slave, so it calls
replicationSetMaster() that will in turn call
replicationCacheMasterUsingMyself(): before this commit, this call would
overwrite the old cached master, creating a memory leak.
2019-10-10 10:23:34 +02:00
Guy Benoish 2fae0192e8 Fix usage of server.stream_node_max_* 2019-10-10 09:47:48 +02:00
Daniel Dai 98600c9a11 update typo 2019-10-09 14:15:31 -04:00
omg-by 1a292e0634 there should is AUTH && HELLO non authenticated state. 2019-10-10 00:52:26 +08:00
Salvatore Sanfilippo a92921da13 Merge pull request #6430 from jaamison/jj/unstable
stream.h: fix typo
2019-10-08 17:40:47 +02:00
Salvatore Sanfilippo 17bf0b25c1 Merge pull request #6427 from oranagra/static-analyzer
fix issues found by a static analyzer
2019-10-08 17:10:57 +02:00
antirez 009862ab7e Geo: output 10 chars of geohash, not 11.
This does not limit the actual precision, because the last digit bits were
garbage, and the shift value became even negative in the last iteration.
2019-10-08 17:09:37 +02:00
Yossi Gottlieb d7f2681a0c TLS: Improve CA certifiate configuration options.
This adds support for explicit configuration of a CA certs directory (in
addition to the previously supported bundle file).  For redis-cli, if no
explicit CA configuration is supplied the system-wide default
configuration will be adopted.
2019-10-08 17:58:50 +03:00
Yossi Gottlieb 61733ded14 TLS: Configuration options.
Add configuration options for TLS protocol versions, ciphers/cipher
suites selection, etc.
2019-10-07 21:07:27 +03:00
Oran Agra 6b6294807c TLS: Implement support for write barrier. 2019-10-07 21:06:30 +03:00
Oran Agra 5a47794606 diskless replication rdb transfer uses pipe, and writes to sockets form the parent process.
misc:
- handle SSL_has_pending by iterating though these in beforeSleep, and setting timeout of 0 to aeProcessEvents
- fix issue with epoll signaling EPOLLHUP and EPOLLERR only to the write handlers. (needed to detect the rdb pipe was closed)
- add key-load-delay config for testing
- trim connShutdown which is no longer needed
- rioFdsetWrite -> rioFdWrite - simplified since there's no longer need to write to multiple FDs
- don't detect rdb child exited (don't call wait3) until we detect the pipe is closed
- Cleanup bad optimization from rio.c, add another one
2019-10-07 21:06:30 +03:00
Yossi Gottlieb b087dd1db6 TLS: Connections refactoring and TLS support.
* Introduce a connection abstraction layer for all socket operations and
integrate it across the code base.
* Provide an optional TLS connections implementation based on OpenSSL.
* Pull a newer version of hiredis with TLS support.
* Tests, redis-cli updates for TLS support.
2019-10-07 21:06:13 +03:00
Jamison Judge 3ef6b79c80 stream.h: fix typo 2019-10-07 11:01:01 -07:00
antirez c65347ab17 LOLWUT: version 6: change text message & credits. 2019-10-07 19:19:38 +02:00
antirez c448f6fe55 LOLWUT: version 6: give credits. 2019-10-07 18:57:25 +02:00
antirez 58cdc054e3 LOLWUT: version 6: improve rithm of the image. 2019-10-07 18:41:29 +02:00
antirez c10889150e LOLWUT: version 6 initial output. May change a bit. 2019-10-07 18:24:34 +02:00
charsyam bea0384f5d fix type salves to slaves 2019-10-07 23:48:11 +09:00
antirez f019f28e7b LOLWUT: version 6 initial concept. 2019-10-07 12:35:17 +02:00
Oran Agra d1a005ab39 fix issues found by a static analyzer
cluster.c - stack buffer memory alignment
    The pointer 'buf' is cast to a more strictly aligned pointer type
evict.c - lazyfree_lazy_eviction, lazyfree_lazy_eviction always called
defrag.c - bug in dead code
server.c - casting was missing parenthesis
rax.c - indentation / newline suggested an 'else if' was intended
2019-10-07 09:09:32 +03:00
antirez bd60447066 LOLWUT: ability to specify VERSION <ver> option. 2019-10-04 19:19:54 +02:00
antirez 9073d56eec LOLWUT: refactoring + skeleton of LOLWUT 6. 2019-10-04 19:19:48 +02:00
Oran Agra 0f1969f16f trim the double implementation of jemalloc purge 2019-10-04 14:22:13 +03:00
Oran Agra bf759cc9c3 Merge remote-tracking branch 'antirez/unstable' into jemalloc_purge_bg 2019-10-04 13:53:40 +03:00
antirez ee1cef189f Minor aesthetic changes to #6419. 2019-10-04 12:00:41 +02:00
Salvatore Sanfilippo 264778af0a Merge pull request #6419 from oranagra/lua_timeout_sha
On LUA script timeout, print the script SHA to the log
2019-10-04 11:58:05 +02:00
Salvatore Sanfilippo c326f28833 Merge pull request #6308 from oranagra/repl-diskless-load-config
fix error handling on config parsing of repl-diskless-load
2019-10-04 11:56:08 +02:00
antirez 3eaff2941e Modules: RM_Call(): give pointer to documentation. 2019-10-04 11:46:53 +02:00
antirez 2a81e49dde Modules: RM_Call/Replicate() ability to exclude AOF/replicas. 2019-10-04 11:44:53 +02:00
antirez 1b8b8c029f Modules: add RM_Replicate() Tcl test file & fix the module. 2019-10-03 18:44:50 +02:00
antirez 55a3da87f8 Modules: RM_Replicate() test with threads. 2019-10-03 13:23:53 +02:00
antirez 45cd8e03ca Modules: RM_Replicate() test module: initial implementation. 2019-10-03 13:06:13 +02:00
antirez 1bca62c4b7 Modules: RM_Replicate() in thread safe contexts. 2019-10-03 11:03:46 +02:00
antirez e938bbc543 Modules: implement RM_Replicate() from async callbacks. 2019-10-03 10:56:37 +02:00
antirez c549513acd Modules: handle propagation when ctx is freed. Flag modules commands ctx. 2019-10-02 18:33:40 +02:00
Oran Agra 2e19b94113 RED-31295 - redis: avoid race between dlopen and thread creation
It seeems that since I added the creation of the jemalloc thread redis
sometimes fails to start with the following error:

Inconsistency detected by ld.so: dl-tls.c: 493: _dl_allocate_tls_init: Assertion `listp->slotinfo[cnt].gen <= GL(dl_tls_generation)' failed!

This seems to be due to a race bug in ld.so, in which TLS creation on the
thread, collide with dlopen.

Move the creation of BIO and jemalloc threads to after modules are loaded.

plus small bugfix when trying to disable the jemalloc thread at runtime
2019-10-02 15:39:44 +03:00
antirez 758b39be99 Speedup INFO server section. 2019-10-02 11:30:20 +02:00
antirez f49f0a6f72 SDS: make sdscatfmt() faster by pre-allocating a bit. 2019-10-02 11:30:20 +02:00
Oran Agra 98426e9886 On LUA script timeout, print the script SHA to the log
since the slowlog and other means that can help you detect the bad script
are only exposed after the script is done. it might be a good idea to at least
print the script name (sha) to the log when it timeouts.
2019-10-02 08:40:35 +03:00
John Sully 0a0c2146c8 Bump version 2019-10-01 17:53:48 -04:00
antirez 40acb4412d GEORADIUS reply: fix of the previous fix about #6417. 2019-10-01 19:18:55 +02:00
Salvatore Sanfilippo 2e2fe98f9c Merge pull request #6270 from oranagra/modules_info
Extend modules API to allow modules report to redis INFO
2019-10-01 18:02:33 +02:00
antirez 3281ebb495 Fix GEORADIUS replies broken after RESP3 introduction.
This commit fixes #6417.
2019-10-01 10:38:56 +02:00
antirez beebb3c013 Merge branch 'hashed-pass-opcode' into unstable 2019-10-01 10:33:04 +02:00
Oran Agra 1b4f888109 Use sdscatfmt instead of sdscatprintf in module info
sdscatfmt is faster
2019-09-30 21:13:13 +03:00
Oran Agra d5c14c70b7 Merge remote-tracking branch 'antirez/unstable' into modules_info 2019-09-30 20:47:35 +03:00
antirez a9628142fd Clarify a comment about memory total_frag field. 2019-09-30 18:37:59 +02:00
antirez 09041b9359 ACLs: change hashed passwords opcode to also remove them.
Related to PR #6405
2019-09-30 18:28:45 +02:00
Madelyn Olson ea7c3fe7fd Allowed passing in of password hash and fixed config rewrite 2019-09-30 17:57:49 +02:00
Salvatore Sanfilippo eb2114b9cd Merge pull request #6411 from nikhilajayk/unstable
Added cluster host and protected mode variables to the create cluster script.
2019-09-30 17:46:09 +02:00
Salvatore Sanfilippo 2478b47b89 Merge pull request #6304 from dbenders/fix-issue-6141
Fix issue #6141: cluster rebalance does not support binary key
2019-09-30 17:43:58 +02:00
antirez 6a2831c24a Fix comments aesthetics. 2019-09-30 17:17:36 +02:00
antirez 23f5cb4233 Change a bit the style of #6385. 2019-09-30 10:58:15 +02:00
Salvatore Sanfilippo ad45d7e407 Merge pull request #6385 from filipecosta90/perf-reply-ss-error
Improve performance of RM_ReplyWithSimpleString and RM_ReplyWi…
2019-09-30 10:57:05 +02:00
John Sully 66cef6646c Merge branch 'unstable' into RELEASE_5 2019-09-30 00:28:12 -04:00
antirez 916c4e5d86 moduleRDBLoadError(): io->ctx may be NULL.
The correct way to access the module about a given IO context is to
deference io->type->module, since io->ctx is only populated if the user
requests an explicit context from an IO object.
2019-09-27 18:42:38 +02:00
antirez 8b5848a4f5 Fix memory leak in moduleLoadFromQueue(). 2019-09-27 18:33:21 +02:00
antirez b394817754 Fix memory leak in RM_UnregisterCommandFilter(). 2019-09-27 18:33:05 +02:00
antirez 0a07f8ffee TerminateModuleForkChild(): fix function prototype. 2019-09-27 12:23:07 +02:00
antirez 721d3c9e0c TerminateModuleForkChild(): move safety checks there.
We don't want that the API could be used directly in an unsafe way,
without checking if there is an active child. Now the safety checks are
moved directly in the function performing the operations.
2019-09-27 12:17:47 +02:00
antirez de1f82aa33 Function renamed hasForkChild() -> hasActiveChildProcess(). 2019-09-27 12:03:09 +02:00
antirez 82845f8d04 TerminateModuleForkChild(): use wait4 for safety.
In theory currently there is only one active child, but the API may
change or for bugs in the implementation we may have several (it was
like that for years because of a bug). Better to wait for a specific
pid and avoid consuing other pending children information.
2019-09-27 11:59:58 +02:00
antirez beb1356b5b Improve error message in BGSAVE. 2019-09-27 11:59:37 +02:00
antirez ae3ef964c1 Modules fork: improve SIGUSR1 handling, fix include.
We can't expect SIGUSR1 to have any specific value range, so let's
define an exit code that we can handle in a special way.
This also fixes an #include <wait.h> that is not standard.
2019-09-27 11:39:45 +02:00
Salvatore Sanfilippo b7c33af4a5 Merge pull request #6247 from oranagra/modules_fork
Module API for Forking
2019-09-27 11:24:45 +02:00
Salvatore Sanfilippo 6129758558 Merge branch 'unstable' into modules_fork 2019-09-27 11:24:06 +02:00
nikhilajayk 6578119a23 Added cluster host and protected mode variables 2019-09-26 21:51:49 +05:30
antirez fddc4757c8 BGREWRITEAOF: improve the generic error message. 2019-09-26 16:14:21 +02:00
Oran Agra 83e87bac76 Fix lastbgsave_status, when new child signal handler get intended kill
And add a test for that.
2019-09-26 15:16:34 +03:00
antirez b7b23bdfb8 INFO: more info about loaded modules.
Related to #6024.
2019-09-26 12:18:55 +02:00
Salvatore Sanfilippo c1ea6175c5 Merge pull request #6024 from itamarhaber/info_modules
Adds a "Modules" section to `INFO`
2019-09-26 11:58:52 +02:00
Salvatore Sanfilippo 959fb5cf68 Merge pull request #6235 from oranagra/module_rdb_load_errors
Allow modules to handle RDB loading errors.
2019-09-26 11:52:42 +02:00
Salvatore Sanfilippo b0a90d8fa8 Merge pull request #4980 from yossigo/rm-assert
Add RedisModule_Assert() API call.
2019-09-26 11:39:29 +02:00
filipecosta90 af15b285fa [fix] fixed the un-refactor bug. 2019-09-25 17:28:42 +01:00
Salvatore Sanfilippo 388bc55344 Merge pull request #6354 from jaredzhu/aarch64-devel
Add aarch64 uc_mcontext debug dump info
2019-09-25 18:14:24 +02:00
antirez b3d6cb268a Modify #6401 changes to fit 80 cols. 2019-09-25 18:08:11 +02:00
Salvatore Sanfilippo f3aaf2b4d8 Merge pull request #6401 from valentinogeron/fix-discard-during-oom
DISCARD should not fail during OOM
2019-09-25 18:07:25 +02:00
Salvatore Sanfilippo f6cf08e582 Merge pull request #6402 from mieko/unstable
Seed SipHash with 128-bit key
2019-09-25 18:03:39 +02:00
antirez 8a531cedb0 ACL: fix ##6408, default user state affecting all the connections. 2019-09-25 17:45:05 +02:00
Mike A. Owens 0a4d2bbd9c Seed SipHash with 128-bit key
SipHash expects a 128-bit key, and we were indeed generating 128-bits,
but restricting them to hex characters 0-9a-f, effectively giving us
only 4 bits-per-byte of key material, and 64 bits overall.

Now, we skip the hex conversion and supply 128 bits of unfiltered
random data.
2019-09-23 19:24:09 -04:00
filipecosta90 733280d9cb [fix] un-refactor the code. [perf] replyWithStatus now makes usage of addReplyProto 2019-09-23 23:45:31 +01:00
antirez eda703ab28 redis-cli: support for ACL style user/pass AUTH. 2019-09-23 19:57:13 +02:00
antirez b21dd082c3 redis-cli: CSV and RAW target for more RESP3 types. 2019-09-23 19:36:06 +02:00
antirez cc108057ff redis-cli: ability to start a session in RESP3 mode. 2019-09-23 17:55:05 +02:00
antirez 8ea185ea30 redis-cli: AUTH can now have 3 arguments as well. 2019-09-23 17:47:36 +02:00
antirez bb7546c913 RESP3: varbatim handling for other redis-cli outputs. 2019-09-23 17:42:57 +02:00
antirez 5e399d5d33 hiredis udpated (RESP3 WIP). 2019-09-23 17:41:04 +02:00
antirez 264708427b RESP3: implementation of verbatim output with TTY target. 2019-09-23 17:39:42 +02:00
valentino 7a73b7f168 DISCARD should not fail during OOM
discard command should not fail during OOM, otherwise client MULTI state
will not be cleared.
2019-09-22 09:22:53 +03:00
antirez c3899720f6 hiredis updated to master version. 2019-09-20 20:08:38 +02:00
David Carlier 5a8a005026 Adding AnonHugePages case + comments 2019-09-20 11:01:36 +01:00
antirez 2e4fa7bb48 Make config.c always_replicate_commands more uniform.
Better if it resembles the other similar code paths.
2019-09-20 11:46:35 +02:00
antirez 1dd36347ed Merge branch 'unstable' of github.com:/antirez/redis into unstable 2019-09-20 11:45:02 +02:00
antirez 630638dcde Remove redundant statement in config.c.
Thanks to @guybe7 for spotting the error in the original PR I merged.
2019-09-20 11:44:32 +02:00
Salvatore Sanfilippo f135d6295b Merge pull request #6367 from Dolondro/unstable
Spelling: Typo fixes for API documentation
2019-09-20 11:33:58 +02:00
Salvatore Sanfilippo 3976f4c88e Merge pull request #6368 from suntw2015/unstable
fix rdb function rdbLoadIntegerObject comment
2019-09-20 11:33:27 +02:00
Salvatore Sanfilippo d0050db236 Merge pull request #6389 from ocadaruma/patch-1
Fix typos in hyperloglog.c
2019-09-20 11:24:07 +02:00
Salvatore Sanfilippo 975eb73145 Merge pull request #6400 from mors741/unstable
Fix compilation error
2019-09-20 11:19:56 +02:00
antirez 0a146a8be4 Add useless break for uniformity / future protection. 2019-09-20 11:19:07 +02:00
Salvatore Sanfilippo bd90c99f83 Merge pull request #6398 from trevor211/fixRESP3ToLuaBool
RESP3: Fix function redisProtocolToLuaType about RESP3->Lua bools.
2019-09-20 11:15:20 +02:00
Salvatore Sanfilippo f42ac40d7c Merge pull request #6397 from trevor211/fixConfigWrongFormatHandling
Fix bad handling of unexpected yesno option
2019-09-20 11:14:38 +02:00
Evgeny a1a65d238c Fix compilation error 2019-09-20 01:11:20 -07:00
WuYunlong 30a3644e64 RESP3: Fix function redisProtocolToLuaType about RESP3->Lua bools. 2019-09-20 08:37:23 +08:00
WuYunlong 74e3a622a2 Fix bad handling of unexpected option while loading config "lua-replicate-commands". 2019-09-20 08:14:36 +08:00
John Sully 4589e861ab Fix issue where AOF events are posted to the wrong event loop and not properly cleaned up 2019-09-19 15:40:09 -04:00
antirez 474a923160 RESP3: Use verbatim in LATENCY subcommands. 2019-09-18 18:53:22 +02:00
antirez cb384127d1 RESP3: Use verbatim in CLIENT LIST. 2019-09-18 18:52:13 +02:00
antirez dd2f695d7e RESP3: Use verbatim in CLUSTER subcommands. 2019-09-18 18:51:15 +02:00
antirez 1b3cb3b0de RESP3: Use verbatim in MEMORY subcommands. 2019-09-18 18:48:14 +02:00
antirez ff9a5d231b RESP3: Use verbatim in DEBUG HTSTATS / HTSTATS-KEY. 2019-09-18 18:46:11 +02:00
antirez aca5482fbf RESP3: Use verbatim in INFO output. 2019-09-18 18:33:13 +02:00
antirez 68519b7034 RESP3: Verbatim conversion test in the LOLWUT command.
redis-cli is currently not able to handle it after going in RESP3 mode,
because of hiredis limitations.
2019-09-18 18:28:51 +02:00
antirez 8ea4bdd91d RESP3: Lua double -> RESP3 conversion. 2019-09-17 19:26:46 +02:00
antirez 89f929b12a RESP3: RESP3 double -> Lua conversion. 2019-09-17 19:20:30 +02:00
antirez 19aba4ac78 RESP3: double -> human readable conversion. 2019-09-17 19:08:33 +02:00
antirez e8e30bc402 RESP3: bool and null values in RESP -> human readable conversion. 2019-09-17 18:57:24 +02:00
Okada Haruki c1cc5ca767 Fix typo 2019-09-17 06:18:01 +09:00
antirez f01f0c02d1 RESP3: convert RESP3 null as Lua nil. Implement RESP3->Lua bools. 2019-09-16 18:36:16 +02:00
antirez 6931004969 RESP3: change behavior of Lua returning true/false for RESP3.
Here we introduce a change in the way we convert values from Lua to
Redis when RESP3 is selected: this is possible without breaking the fact
we can return directly what a command returned, because there is no
Redis command in RESP2 that returns true or false to Lua, so the
conversion in the case of RESP2 is totally arbitrary. When a script is
written selecting RESP3 from Lua, it totally makes sense to change such
behavior and return RESP3 true/false when Lua true/false is returned.
2019-09-16 18:18:17 +02:00
antirez 2cc4d0286c RESP3: implement new NULL representation parsing in Lua. 2019-09-16 17:49:47 +02:00
antirez ca81d49006 RESP3: handle set Lua -> Redis conversion. 2019-09-16 12:19:19 +02:00
antirez c792504fab RESP3: handle map Lua -> Redis conversion. 2019-09-16 12:15:39 +02:00
antirez dad38c19c8 RESP3: report set/map as nested tables to Lua. 2019-09-16 11:49:42 +02:00
filipecosta90 4a30a26f8f [add] improved performance of RM_ReplyWithSimpleString and RM_ReplyWithError by making usage addReplyProto instead of addReplySds 2019-09-15 21:16:30 +01:00
David Carlier 819a661be5 Getting region date per process in Darwin 2019-09-15 14:05:00 +01:00
antirez 888efc1b36 RESP3: Lua parsing should depend on lua client, not lua caller.
We want all the scripts to run in RESP2 mode by default. It's up to the
caller to switch to V3 using redis.setresp() if it is really needed.
This way most scripts written for past Redis versions will continue to
work with Redis >= 6 even if the client is in RESP3 mode.
2019-09-13 19:38:39 +02:00
antirez 26b6c697d3 RESP3: Lua debugger support for printing sets and maps. 2019-09-13 19:19:10 +02:00
antirez 93c52ff5ff RESP3: implement lua.setresp(). 2019-09-13 19:01:39 +02:00
antirez b170a01fa7 ACL: store hashed passwords in memory.
Note that this breaks API compatibility with Redis < 6:

    CONFIG GET requirepass

Will no longer return a cleartext password as well, but the SHA256 hash
of the password set.
2019-09-12 12:54:57 +02:00
antirez ae5054b476 ACL: SHA256 based password hashing function implemented. 2019-09-12 12:33:22 +02:00
antirez 9d2ecf6be3 ACL: add slightly modified version of sha256.c for password hashing.
memory.h include removed, types substituted with stdint types.
2019-09-12 12:21:37 +02:00
antirez f4d37173fe ACL: protect MULTI/EXEC transactions after rules change. 2019-09-11 19:42:10 +02:00
John Sully 71d5a10621 bump version 2019-09-06 13:31:40 -04:00
John Sully 50a6a3ca38 Fix issue where Active Replicas were commiting data to the wrong database under load 2019-09-06 13:30:15 -04:00
antirez ae8cab3296 Merge branch 'unstable' of github.com:/antirez/redis into unstable 2019-09-06 12:24:40 +02:00
antirez a092f20d87 handleClientsBlockedOnKeys() refactoring. 2019-09-06 12:24:26 +02:00
suntiawnen 781f3fd8be fix rdb function rdbLoadIntegerObject comment 2019-09-06 12:01:44 +08:00
Doug Nelson cc8f06ece9 Typo fixes in API documentation 2019-09-05 16:25:06 +01:00
Salvatore Sanfilippo 86aade9a02 Merge pull request #6364 from oranagra/fix_module_aux_when
Fix to module aux data rdb format for backwards compatibility with old check-rdb
2019-09-05 13:30:26 +02:00
Oran Agra 3bfcae247a Fix to module aux data rdb format for backwards compatibility with old check-rdb
When implementing the code that saves and loads these aux fields we used rdb
format that was added for that in redis 5.0, but then we added the 'when' field
which meant that the old redis-check-rdb won't be able to skip these.
this fix adds an opcode as if that 'when' is part of the module data.
2019-09-05 14:11:37 +03:00
antirez 89ad0ca566 Fix handleClientsBlockedOnKeys() names in comments. 2019-09-05 13:05:57 +02:00
antirez 12c56a8e75 redis-cli: always report server errors on read errors.
Before this commit we may have not consumer buffers when a read error is
encountered. Such buffers may contain errors that are important clues
for the user: for instance a protocol error in the payload we send in
pipe mode will cause the server to abort the connection. If the user
does not get the protocol error, debugging what is happening can be a
nightmare.

This commit fixes issue #3756.
2019-09-04 17:55:46 +02:00
antirez f7090f43ad AOF: be future-proof and close the file pointer.
Currently useless but we release the fake client, so better to do a full
cleanup. Thanks to @TomMD reporting this in #6353.
2019-09-04 17:20:37 +02:00
zhudacai 00228490 66a5c1cd7e The aarch64 architecture is support normal memory unaligned accesses,
so add the UNALIGNED_LE_CPU to the aarch64 .
2019-09-04 12:14:25 +00:00
antirez b2e10131c0 Rio: fix flag name, function is never used btw.
Thanks to @tnclong for reporting the problem.
2019-09-04 13:01:07 +02:00
antirez 06d490342f RESP3: fix cases of NULL reported instead of empty aggregate. 2019-09-02 12:50:47 +02:00
antirez b12ccc840e Cluster: abort loading nodes data if vars arguments are unbalanced.
See for reference PR #6337. Thanks to @git-hulk for spotting this.
2019-09-02 11:41:20 +02:00
antirez e57067b677 More strict checks and better comments in flushSlaveOutputBuffers().
Related to #6296.
2019-08-31 14:46:24 +02:00
antirez 2303ba1441 Merge branch 'unstable' of github.com:/antirez/redis into unstable 2019-08-31 14:41:28 +02:00
antirez ca5ccc58c5 Improve comment in flushSlavesOutputBuffers(). 2019-08-31 14:40:09 +02:00
Salvatore Sanfilippo 16435e02bf Merge pull request #6296 from soloestoy/flushSlavesOutputBuffers-bugfix
networking: flushSlavesOutputBuffers bugfix
2019-08-31 14:39:33 +02:00
zhudacai 00228490 d7c25806c0 src/debug.c do not support aarch64 dump utcontext, so add the context
of aarch64.
The content comes from the definition of the sigcontext and tested on
my aarch64 server.

sigcontext defined at the linux kernel code:
arch/arm64/include/uapi/asm/sigcontext.h
2019-08-31 07:47:11 +00:00
jared 5cbc73c657 Merge pull request #1 from antirez/unstable
merge redis
2019-08-30 17:25:56 +08:00
John Sully f83a89f82a Acquire the lock for modules that don't acquire it before calling like they are supposed to 2019-08-29 15:41:03 -04:00
John Sully bf905a78f0 Bump version 2019-08-27 15:32:24 -04:00
John Sully fad9483fc9 Fix race condition in PUB/SUB and other async reply commands where the client can be freed before our handler is executed on the client thread. When this occurs the client pointer is dangling 2019-08-27 15:31:47 -04:00
Oran Agra 78bbb9b58d Modlue fork is killed when the parent exists 2019-08-25 10:11:48 +03:00
Salvatore Sanfilippo d5536e0480 Merge pull request #6307 from oranagra/register_rm_reply_with_cstring
RM_ReplyWithCString was missing registration
2019-08-22 20:02:17 +02:00
Oran Agra 61853ad8de Module INFO, support default section for simple modules 2019-08-18 10:01:57 +03:00
Oran Agra 1d6e5dc4dc Module INFO, add support for dict fields, rename API to have common prefix 2019-08-18 09:41:45 +03:00
Oran Agra 1026d2caf8 fix error handling on config parsing of repl-diskless-load 2019-08-08 14:53:36 +03:00
Oran Agra 0a97149dec RM_ReplyWithCString was missing registration 2019-08-07 13:06:11 +03:00
Diego Bendersky b27f388344 pass len to alloc in clusterManagerMigrateKeysInReply 2019-08-05 17:35:50 -03:00
antirez 0e9b5adbd3 Replication: clarify why repl_put_online_on_ack exists at all. 2019-08-05 17:38:15 +02:00
antirez 5e0faf4959 tracking_collisions.c: sha1 + crc64 implemented. 2019-08-02 20:24:27 +02:00
antirez a368209b1d tracking_collisions.c: initial skeleton.
... of a program to just test the hashing functions collisions on a 24
bit output with strings that are very likely Redis key names, and names
of a kind that are particularly prone to collisions.
2019-08-02 20:13:21 +02:00
zhaozhao.zz d3f7eb0c87 networking: flushSlavesOutputBuffers bugfix 2019-08-02 17:17:19 +08:00
antirez 583933e2d6 Fix regression causing EXEC to appear in the slow log.
This was recently introduced with PR #6266.
2019-07-31 19:05:20 +02:00
antirez 2a8a63af44 Make EMBSTR case of #6261 more obvious. 2019-07-31 12:03:10 +02:00
Salvatore Sanfilippo f899c31fbc Merge pull request #6261 from chendq8/robj
make memory usage consistent of robj with OBJ_ENCODING_INT
2019-07-31 11:54:11 +02:00
Salvatore Sanfilippo fc5c2052b5 Merge pull request #6266 from madolson/dev-unstable-hide-auth-and-hello
Hide HELLO and AUTH from slowlog and monitor
2019-07-31 11:12:46 +02:00
antirez e1b13b4d39 Merge branch 'unstable' of github.com:/antirez/redis into unstable 2019-07-31 10:37:17 +02:00
antirez b2079cb508 HyperLogLog: fix the fix of a corruption bug. 2019-07-31 10:36:23 +02:00
Salvatore Sanfilippo f0c10d4360 Merge pull request #6277 from devnexen/fbsd_get_rss
Updating resident memory request impl on FreeBSD.
2019-07-30 18:24:38 +02:00
Oran Agra 40c4183196 Log message when modules prevent diskless-load 2019-07-30 16:32:58 +03:00
Oran Agra 07b1abab95 Add test for module diskless short reads 2019-07-30 15:20:03 +03:00
Oran Agra e5187ad2ae Merge remote-tracking branch 'oss/unstable' into module_rdb_load_errors 2019-07-30 15:14:08 +03:00
Oran Agra 4339706e07 Avoid diskelss-load if modules did not declare they handle read errors 2019-07-30 15:11:57 +03:00
Salvatore Sanfilippo f42846e8c7 Merge pull request #6264 from oranagra/modules_api_aux_rdb
Implement module api for aux data in rdb
2019-07-30 11:50:44 +02:00
Salvatore Sanfilippo ce72cac0f8 Merge pull request #6269 from wubostc/patch-1
Reduce the calling stack
2019-07-30 11:41:48 +02:00
antirez e4b3c8bbc3 emptyDbGeneric(): call signalFlushDb() before deleting the keys.
This was broken since a refactoring performed recently by myself.
2019-07-30 11:20:54 +02:00
Salvatore Sanfilippo cc8125a779 Merge pull request #6282 from JohnSully/hll_corruption
Fix HLL corruption bug
2019-07-30 10:24:28 +02:00
John Sully d659654f53 Fix HLL corruption bug 2019-07-29 18:11:52 -04:00
John Sully 6b6e6f6c1e Fix crash in RediSearch 2019-07-29 15:08:57 -04:00
David Carlier f1c6c658ac Updating resident memory request impl on FreeBSD. 2019-07-28 14:33:57 +01:00
antirez 505a855000 Merge branch 'unstable' of github.com:/antirez/redis into unstable 2019-07-25 19:18:26 +02:00
antirez 0714581e0c Mark diskless loads as experimental in redis.conf. 2019-07-25 19:17:58 +02:00
Salvatore Sanfilippo 5591a14c0a Merge pull request #6248 from oranagra/module-log
RM_Log - add support for logging without a context or context without module
2019-07-25 18:48:46 +02:00
John Sully b0a0f03b96 RREPLAY failures should be logged 2019-07-24 22:49:55 -04:00
John Sully f1982ca63d Issue #64 RREPLAY isn't binary safe. Add fix and test. 2019-07-24 22:31:44 -04:00
Oran Agra e91d9a6fff Extend modules API to allow modules report to redis INFO
this implements #6012
2019-07-24 12:58:15 +03:00
antirez fde5e737dc Merge branch 'unstable' of github.com:/antirez/redis into unstable 2019-07-24 11:45:19 +02:00
antirez e53a26b5d8 Client side caching: document tracking-table-max-fill in redis.conf. 2019-07-24 11:45:10 +02:00
antirez 0b780a5477 Example redis.conf: stay under 80 cols. 2019-07-24 11:38:11 +02:00
antirez 32efd9adf8 Client side caching: config option for table fill rate. 2019-07-24 11:35:01 +02:00
antirez a67d0411e7 Remove experimental warning from diskless replication. 2019-07-24 11:33:53 +02:00
wubostc 48f4cfb601 Reduce the calling stack 2019-07-24 16:22:26 +08:00
Salvatore Sanfilippo dbee7ca007 Merge pull request #6265 from madolson/dev-unstable-remove-create-object
Removed unnecessary creation of Redis objects
2019-07-23 17:05:10 +02:00
antirez c98e7717bb Client side caching: show tracking slots usage in INFO. 2019-07-23 11:02:14 +02:00
antirez 9268493e8d Client side caching: implement full slot limit function. 2019-07-23 10:57:22 +02:00
Salvatore Sanfilippo 47ce1cebeb Merge pull request #6267 from soloestoy/opt-claim-trackingtable
Client side caching: do not reclaim tracking table if it's empty
2019-07-23 09:43:41 +02:00
zhaozhao.zz 9012e587ca Client side caching: do not reclaim tracking table if it's empty 2019-07-23 15:25:00 +08:00
Madelyn Olson 7d21754710 Hide HELLO and AUTH from slowlog and monitor 2019-07-22 22:53:15 -07:00
Madelyn Olson 8fe173452a Removed unecessary creation of Redis objects 2019-07-22 22:52:16 -07:00
Oran Agra 3b6aeea44c Implement module api for aux data in rdb
Other changes:
* fix memory leak in error handling of rdb loading of type OBJ_MODULE
2019-07-22 21:15:33 +03:00
antirez c41f94d2a3 Client side caching: split invalidation into key / slot. 2019-07-22 18:59:53 +02:00
antirez 09c06698e9 Client side caching: redis-cli ability to enable tracking.
This is extremely useful in order to simulate an high load of requests
about different keys, and force Redis to track a lot of informations
about several clients, to simulate real world workloads.
2019-07-22 18:45:47 +02:00
antirez 436d02dd7b Move signalFlushedDb() into a better place.
Now that the call also invalidates client side caching slots, it is
important that after an internal flush operation we both send the
notifications to the clients and, at the same time, are able to reclaim
the memory of the tracking table. This may even fix a few edge cases
related to MULTI/EXEC + WATCH during resync, not sure, but in general
looks more correct.
2019-07-22 12:37:47 +02:00
antirez 7d3992a80a Client side caching: reclaim the tracking table on FLUSHALL. 2019-07-22 12:33:49 +02:00
antirez 842b44dc46 Client side caching: call the invalidation functions always.
Otherwise what happens is that the tracking table will never get garbage
collected if there are no longer clients with tracking enabled.
Now the invalidation function immediately checks if there is any table
allocated, otherwise it returns ASAP, so the overhead when the feature
is not used should be near zero.
2019-07-22 12:29:54 +02:00
antirez f850201c64 Client side caching: don't hash the key if not needed. 2019-07-22 12:10:51 +02:00
antirez e00442eb13 Clinet side caching: take count of used caching slots. 2019-07-22 11:47:44 +02:00
chendianqiang 004c26ec28 make memory usage consistent of robj with OBJ_ENCODING_INT 2019-07-22 17:45:30 +08:00
Oran Agra d7d028a7a7 Allow modules to handle RDB loading errors.
This is especially needed in diskless loading, were a short read could have
caused redis to exit. now the module can handle the error and return to the
caller gracefully.

this fixes #5326
2019-07-21 18:19:32 +03:00
John Sully ff682d7902 Modules must have execute permissions to load 2019-07-19 15:28:31 -04:00
antirez bc5cb168f5 RDB: fix MODULE_AUX loading by continuing to next opcode.
Thanks to @JohnSully for noticing this problem.
2019-07-19 11:12:46 +02:00
John Sully 70534846d1 Modules must have execute permissions to load 2019-07-19 01:37:34 -04:00
John Sully 25e18a7de9 Release version 5 2019-07-18 20:50:40 -04:00
antirez 5f450e4928 RDB: make sure to abort on LZF encoding error. 2019-07-18 18:59:38 +02:00
antirez bd0f06c18c RDB: handle encoding errors with rdbExitReportCorruptRDB().
Without such change, the diskless replicas, when loading RDB files from
the socket will not abort when a broken RDB file gets loaded. This is
potentially unsafe, because right now Redis is not able to guarantee
that encoding errors are safe from the POV of memory corruptions (for
instance the LZF library may not be safe against untrusted data?) so
better to abort when the RDB file we are going to load is corrupted.

Instead I/O errors are still returned to the caller without aborting,
so that in case of short read the diskless replica can try again.
2019-07-18 18:51:45 +02:00
Salvatore Sanfilippo 41ed85bd26 Merge pull request #6252 from soloestoy/tracking-flushdb
Tracking flushdb
2019-07-18 16:41:37 +02:00
antirez 947319cadd RDB: update rdbLoadRio comment about EOF condition. 2019-07-18 12:37:55 +02:00
zhaozhao.zz f59119aad6 Client side caching: filter clients untracking 2019-07-18 16:46:19 +08:00
antirez 42b6305964 RDB: try to make error handling code more readable. 2019-07-17 17:30:02 +02:00
antirez 6b72b04a37 Rio: when in error condition avoid doing the operation. 2019-07-17 16:46:22 +02:00
antirez 5189db3d81 RDB: modify rdbReportError() var name for clarity. 2019-07-17 16:46:22 +02:00
antirez 48d91cf4cc Rio: remember read/write error conditions. 2019-07-17 16:46:22 +02:00
Oran Agra c56b4ddc6f prevent diskless replica from terminating on short read
now that replica can read rdb directly from the socket, it should avoid exiting
on short read and instead try to re-sync.

this commit tries to have minimal effects on non-diskless rdb reading.
and includes a test that tries to trigger this scenario on various read cases.
2019-07-17 16:46:22 +02:00
Oran Agra 56258c6b7d Module API for Forking
* create module API for forking child processes.
* refactor duplicate code around creating and tracking forks by AOF and RDB.
* child processes listen to SIGUSR1 and dies exitFromChild in order to
  eliminate a valgrind warning of unhandled signal.
* note that BGSAVE error reply has changed.

valgrind error is:
  Process terminating with default action of signal 10 (SIGUSR1)
2019-07-17 16:40:24 +03:00
zhaozhao.zz 6191ea90a1 Client side caching: implement trackingInvalidateKeysOnFlush() 2019-07-17 20:33:52 +08:00
Oran Agra 6e07fac405 RM_Log - add support for logging without a context or context without module
for instance detached thread safe contexts, or various callbacks that don't
provide a context.
2019-07-17 15:08:18 +03:00
antirez 241d18d954 Merge branch 'unstable' of github.com:/antirez/redis into unstable 2019-07-17 12:26:38 +02:00
Salvatore Sanfilippo 5827b69861 Merge pull request #6185 from AlexRenCN/unstable
fix readme.md,Redis data types should add `t_stream.c`.
2019-07-16 18:33:55 +02:00
Salvatore Sanfilippo 4b2579a064 Merge pull request #5779 from madolson/dev-unstable-geo-bugfix
Fixed a rounding bug in geo.tcl
2019-07-15 23:41:57 +02:00
Salvatore Sanfilippo 3f1c84751a Merge pull request #6196 from AngusP/resp3
[RESP3, Minor] in networking.c double representation for -infiinity leaves out comma
2019-07-12 13:08:46 +02:00
Salvatore Sanfilippo 6100a82a3a Merge pull request #5738 from bmerry/fix-swapdb-int-overflow
Make dbSwapDatabases take args as long
2019-07-12 12:36:33 +02:00
antirez 54c4e7f86c Vertically compact code in aofWrite(). 2019-07-12 12:18:33 +02:00
antirez 2a99f199bc Merge branch 'unstable' of github.com:/antirez/redis into unstable 2019-07-10 19:48:44 +02:00
antirez dad0e916ff Client side caching: filter clients no longer there. 2019-07-10 19:46:46 +02:00
Salvatore Sanfilippo 7567323071 Merge pull request #6198 from oranagra/skiplist-mem-usage
missing per-skiplist overheads in MEMORY USAGE
2019-07-10 19:43:24 +02:00
antirez c7aaf8db4d Client side caching: implement CLIENT GETREDIR.
This subcommand may simplify the writing of Redis client libraries
using the tracking feature and/or improve observability and debugging
capabilities.
2019-07-10 18:17:10 +02:00
antirez 923e4fb312 Client side caching: add tracking clients in INFO. 2019-07-10 18:08:31 +02:00
antirez 7c9f6ebc8f Diskless replica: fix disklessLoadRestoreBackups() bug. 2019-07-10 12:36:26 +02:00
antirez 3bbb9a1413 Diskless replica: refactoring of DBs backups. 2019-07-10 11:42:26 +02:00
antirez 3300e98ff0 Diskless replica: fix mispelled var name. 2019-07-10 09:34:21 +02:00
antirez dfcbeaf115 Diskless replica: a few aesthetic changes to rio.c 2019-07-08 18:39:59 +02:00
antirez 81b18fa3a0 Diskless replica: a few aesthetic changes to replication.c. 2019-07-08 18:32:47 +02:00
Salvatore Sanfilippo d984732b35 Merge pull request #6193 from oranagra/diskless_slave_refresh3
diskless replication on slave side (don't store rdb to file), plus some other related fixes
2019-07-08 18:10:16 +02:00
Oran Agra 2de544cfcc diskless replication on slave side (don't store rdb to file), plus some other related fixes
The implementation of the diskless replication was currently diskless only on the master side.
The slave side was still storing the received rdb file to the disk before loading it back in and parsing it.

This commit adds two modes to load rdb directly from socket:
1) when-empty
2) using "swapdb"
the third mode of using diskless slave by flushdb is risky and currently not included.

other changes:
--------------
distinguish between aof configuration and state so that we can re-enable aof only when sync eventually
succeeds (and not when exiting from readSyncBulkPayload after a failed attempt)
also a CONFIG GET and INFO during rdb loading would have lied

When loading rdb from the network, don't kill the server on short read (that can be a network error)

Fix rdb check when performed on preamble AOF

tests:
run replication tests for diskless slave too
make replication test a bit more aggressive
Add test for diskless load swapdb
2019-07-08 15:37:48 +03:00
Oran Agra 41c5ebcab6 missing per-skiplist overheads in MEMORY USAGE
these had severe impact for small zsets, for instance ones with just one
element that is longer than 64 (causing it not to be ziplist encoded)
2019-07-04 15:20:49 +03:00
Angus Pearson 593f6656c1 RESP3 double representation for -infinity is ,-inf\r\n, not -inf\r\n 2019-07-02 14:28:48 +01:00
zheng.ren01@mljr.com 2d236d7aec fix readme.md,Redis data types should add t_stream.c. 2019-06-25 18:34:35 +08:00
Oran Agra 09f99c2a92 make redis purge jemalloc after flush, and enable background purging thread
jemalloc 5 doesn't immediately release memory back to the OS, instead there's a decaying
mechanism, which doesn't work when there's no traffic (no allocations).
this is most evident if there's no traffic after flushdb, the RSS will remain high.

1) enable jemalloc background purging
2) explicitly purge in flushdb
2019-06-02 15:33:14 +03:00
Oran Agra 2fec7d9c6c Jemalloc: Avoid blocking on background thread lock for stats.
Background threads may run for a long time, especially when the # of dirty pages
is high.  Avoid blocking stats calls because of this (which may cause latency
spikes).

see https://github.com/jemalloc/jemalloc/issues/1502

cherry picked from commit 1a71533511027dbe3f9d989659efeec446915d6b
2019-06-02 15:27:38 +03:00
Yuan Zhou 412eb67d21 aof: fix assignment for aof_fsync_offset
Signed-off-by: Yuan Zhou <yuan.zhou@intel.com>
2019-05-29 14:48:46 +08:00
Itamar Haber 52686f4866 Adds a "Modules" section to INFO
Fixes #6012.

As long as "INFO is broken", this should be adequate IMO. Once we rework
`INFO`, perhaps into RESP3, this implementation should be revisited.
2019-04-16 22:16:12 +03:00
Guy Korland c1455dc060 Update config.yml 2019-03-15 22:22:06 +02:00
Guy Korland 656cd1a00a Update pull.yml 2019-03-15 21:35:24 +02:00
Guy Korland c2b7f14c56 Update config.yml 2019-03-15 21:30:09 +02:00
Guy Korland 3c05bc561d add pull app 2019-03-15 21:14:15 +02:00
Guy Korland a2ac5c3840 add CI 2019-03-15 21:09:59 +02:00
Yossi Gottlieb 68fd59056b Add RedisModule_Assert() API call. 2019-03-14 15:00:31 +02:00
Jim Brunner 283d6cfd58 Addition of OnUnload function 2019-03-13 16:31:24 +00:00
Madelyn Olson b00e1891c0 Fixed a rounding bug in geo.tcl 2019-01-15 07:26:19 +00:00
Bruce Merry 6b818efeb6 Make dbSwapDatabases take args as long
This prevents an integer overflow bug. Closes #5737.
2018-12-31 11:51:03 +02:00
jem a344c4cfad ignore vscode conf dir 2018-09-18 20:42:09 +08:00
jem 42366959ec update leap year comment when div by 400 2018-09-18 17:04:00 +08:00
Wander Hillen dda8cc1821 More typos 2018-03-16 09:59:17 +01:00
Wander Hillen dcffca0a31 Fix typos, add some periods 2018-03-16 09:59:14 +01:00
406 changed files with 39660 additions and 6747 deletions

No files matched your search

+58
View File
@@ -0,0 +1,58 @@
name: CI
on: [push, pull_request]
jobs:
test-ubuntu-latest:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v2
- name: make
run: |
sudo apt-get update
sudo apt-get -y install uuid-dev libcurl4-openssl-dev
make BUILD_TLS=yes -j2
- name: gen-cert
run: ./utils/gen-test-certs.sh
- name: test-tls
run: |
sudo apt-get -y install tcl8.5 tcl-tls
./runtest --clients 2 --verbose --tls
- name: cluster-test
run: |
./runtest-cluster --tls
- name: sentinel test
run: |
./runtest-sentinel
- name: module tests
run: |
./runtest-moduleapi
build-ubuntu-old:
runs-on: ubuntu-16.04
steps:
- uses: actions/checkout@v1
- name: make -j2
run: |
sudo apt-get update
sudo apt-get -y install uuid-dev libcurl4-openssl-dev
make -j2
build-macos-latest:
runs-on: macos-latest
steps:
- uses: actions/checkout@v2
- name: make
run: make -j2
build-libc-malloc:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v2
- name: make
run: |
sudo apt-get update
sudo apt-get -y install uuid-dev libcurl4-openssl-dev
make MALLOC=libc -j2
+179
View File
@@ -0,0 +1,179 @@
name: Daily
on:
pull_request:
branches:
# any PR to a release branch.
- '[0-9].[0-9]'
schedule:
- cron: '0 0 * * *'
jobs:
test-ubuntu-jemalloc:
runs-on: ubuntu-latest
if: github.repository == 'redis/redis'
timeout-minutes: 14400
steps:
- uses: actions/checkout@v2
- name: make
run: |
sudo apt-get -y install uuid-dev libcurl4-openssl-dev
make
- name: test
run: |
sudo apt-get install tcl8.5
./runtest --accurate --verbose
- name: module api test
run: ./runtest-moduleapi --verbose
- name: sentinel tests
run: ./runtest-sentinel
- name: cluster tests
run: ./runtest-cluster
test-ubuntu-libc-malloc:
runs-on: ubuntu-latest
if: github.repository == 'redis/redis'
timeout-minutes: 14400
steps:
- uses: actions/checkout@v2
- name: make
run: |
sudo apt-get -y install uuid-dev libcurl4-openssl-dev
make MALLOC=libc
- name: test
run: |
sudo apt-get install tcl8.5
./runtest --accurate --verbose
- name: module api test
run: ./runtest-moduleapi --verbose
- name: sentinel tests
run: ./runtest-sentinel
- name: cluster tests
run: ./runtest-cluster
test:
runs-on: ubuntu-latest
if: github.repository == 'redis/redis'
timeout-minutes: 14400
steps:
- uses: actions/checkout@v2
- name: make
run: |
sudo apt-get -y install uuid-dev libcurl4-openssl-dev
make BUILD_TLS=yes -j2
- name: "test (tls)"
run: |
sudo apt-get install tcl8.5 tcl-tls
./utils/gen-test-certs.sh
./runtest --accurate --verbose --tls
- name: test
run: ./runtest --accurate --verbose
- name: module api test (tls)
run: ./runtest-moduleapi --verbose --tls
test-ubuntu-arm:
runs-on: [self-hosted, linux, arm]
steps:
- uses: actions/checkout@v2
- name: make
run: |
sudo apt-get -y install uuid-dev libcurl4-openssl-dev
make -j4
- name: test
run: |
sudo apt-get -y install tcl8.5
./runtest --clients 2 --verbose
- name: module tests
run: |
./runtest-moduleapi
test-valgrind:
runs-on: ubuntu-latest
if: github.repository == 'redis/redis'
timeout-minutes: 14400
steps:
- uses: actions/checkout@v2
- name: make
run: |
sudo apt-get -y install uuid-dev libcurl4-openssl-dev
make valgrind
- name: test
run: |
sudo apt-get install tcl8.5 valgrind -y
./runtest --valgrind --verbose --clients 1
- name: module api test
run: ./runtest-moduleapi --valgrind --verbose --clients 1
test-centos7-jemalloc:
runs-on: ubuntu-latest
if: github.repository == 'redis/redis'
container: centos:7
timeout-minutes: 14400
steps:
- uses: actions/checkout@v2
- name: make
run: |
yum -y install centos-release-scl
yum -y install devtoolset-7
scl enable devtoolset-7 "make"
- name: test
run: |
yum -y install tcl
./runtest --accurate --verbose
- name: module api test
run: ./runtest-moduleapi --verbose
- name: sentinel tests
run: ./runtest-sentinel
- name: cluster tests
run: ./runtest-cluster
test-centos7-tls:
runs-on: ubuntu-latest
if: github.repository == 'redis/redis'
container: centos:7
timeout-minutes: 14400
steps:
- uses: actions/checkout@v2
- name: make
run: |
yum -y install centos-release-scl epel-release
yum -y install devtoolset-7 openssl-devel openssl
scl enable devtoolset-7 "make BUILD_TLS=yes"
- name: test
run: |
yum -y install tcl tcltls
./utils/gen-test-certs.sh
./runtest --accurate --verbose --tls
./runtest --accurate --verbose
- name: module api test
run: |
./runtest-moduleapi --verbose --tls
./runtest-moduleapi --verbose
- name: sentinel tests
run: |
./runtest-sentinel --tls
./runtest-sentinel
- name: cluster tests
run: |
./runtest-cluster --tls
./runtest-cluster
test-macos-latest:
runs-on: macos-latest
if: github.repository == 'redis/redis'
timeout-minutes: 14400
steps:
- uses: actions/checkout@v2
- name: make
run: make
- name: test
run: |
./runtest --accurate --verbose
- name: module api test
run: ./runtest-moduleapi --verbose
- name: sentinel tests
run: ./runtest-sentinel
- name: cluster tests
run: ./runtest-cluster
+22
View File
@@ -0,0 +1,22 @@
name: Endurance
on:
schedule:
- cron: '30 * * * *'
jobs:
test-endurance:
runs-on: octocore
timeout-minutes: 60
steps:
- uses: actions/checkout@v1
- name: make
run: |
sudo apt-get -y install uuid-dev libcurl4-openssl-dev
make -j8
- name: test-multithread (5X)
run: |
sudo apt-get install -y tcl tcl-tls
./runtest --loopn 5 --config server-threads 3 --clients 5 --endurance
+24 -3
View File
@@ -1,9 +1,27 @@
.*.swp
core
*.o
*.xo
*.so
*.d
!**/bash_completion.d
!**/logrotate.d
!**/keydb.service.d
!**/keydb-sentinel.service.d
*.log
dump.rdb
src/keydb-server
**/bin/keydb-server
**/app/keydb-server
*.deb
*.rpm
keydb-pro-server
src/keydb-cli
**/bin/keydb-cli
**/app/keydb-cli
src/keydb-sentinel
**/bin/keydb-sentinel
**/app/keydb-sentinel
redis-benchmark
keydb-benchmark
redis-check-aof
@@ -13,11 +31,8 @@ keydb-check-rdb
redis-check-dump
keydb-check-dump
redis-cli
keydb-cli
redis-sentinel
keydb-sentinel
redis-server
keydb-server
doc-tools
release
misc/*
@@ -33,7 +48,13 @@ src/nodes.conf
deps/lua/src/lua
deps/lua/src/luac
deps/lua/src/liblua.a
tests/tls/*
.make-*
.prerequisites
*.dSYM
Makefile.dep
.vscode/*
.idea/*
.ccls
.ccls-cache/*
compile_commands.json
+3
View File
@@ -0,0 +1,3 @@
[submodule "deps/depot_tools"]
path = deps/depot_tools
url = https://chromium.googlesource.com/chromium/tools/depot_tools.git
-29
View File
@@ -1,29 +0,0 @@
language: generic
os: osx
matrix:
include:
- os: linux
script: make
env: COMPILER_NAME=g++-6 CXX=g++-6 CC=gcc-6
addons:
apt:
packages:
- g++-6
- nasm
- uuid-dev
sources: &sources
- llvm-toolchain-precise-3.8
- ubuntu-toolchain-r-test
- os: linux
script: make MALLOC=libc
env: COMPILER_NAME=clang CXX=clang++-3.8 CC=clang-3.8 CXXFLAGS="-I/usr/include/libcxxabi/" LDFLAGS="-lc++"
addons:
apt:
packages:
- libc++abi-dev
- clang-3.8
- libc++-dev
- libc++abi-dev
- nasm
- uuid-dev
sources: *sources
-58
View File
@@ -1,58 +0,0 @@
{
"files.associations": {
"zmalloc.h": "c",
"stat.h": "c",
"array": "cpp",
"atomic": "cpp",
"*.tcc": "cpp",
"cctype": "cpp",
"chrono": "cpp",
"clocale": "cpp",
"cmath": "cpp",
"condition_variable": "cpp",
"cstdarg": "cpp",
"cstddef": "cpp",
"cstdint": "cpp",
"cstdio": "cpp",
"cstdlib": "cpp",
"cstring": "cpp",
"ctime": "cpp",
"cwchar": "cpp",
"cwctype": "cpp",
"deque": "cpp",
"list": "cpp",
"unordered_map": "cpp",
"vector": "cpp",
"exception": "cpp",
"fstream": "cpp",
"functional": "cpp",
"future": "cpp",
"initializer_list": "cpp",
"iomanip": "cpp",
"iosfwd": "cpp",
"iostream": "cpp",
"istream": "cpp",
"limits": "cpp",
"memory": "cpp",
"mutex": "cpp",
"new": "cpp",
"numeric": "cpp",
"optional": "cpp",
"ostream": "cpp",
"ratio": "cpp",
"scoped_allocator": "cpp",
"sstream": "cpp",
"stdexcept": "cpp",
"streambuf": "cpp",
"string_view": "cpp",
"system_error": "cpp",
"thread": "cpp",
"cinttypes": "cpp",
"tuple": "cpp",
"type_traits": "cpp",
"typeinfo": "cpp",
"utility": "cpp",
"set": "cpp",
"algorithm": "cpp"
}
}
+2859 -11
View File
File diff suppressed because it is too large. Load diff
+1 -1
View File
@@ -1 +1 @@
Please check https://github.com/antirez/redis/issues
Please check https://github.com/redis/redis/issues
+2 -2
View File
@@ -1,5 +1,5 @@
Copyright (c) 2006-2015, Salvatore Sanfilippo
Copyright (C) 2019, John Sully
Copyright (c) 2006-2020, Salvatore Sanfilippo
Copyright (C) 2019-2020, John Sully
All rights reserved.
Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:
-10
View File
@@ -1,10 +0,0 @@
FROM ubuntu:18.04
RUN apt-get update \
&& DEBIAN_FRONTEND=noninteractive apt-get install -qqy \
build-essential nasm autotools-dev autoconf libcurl4-openssl-dev libjemalloc-dev tcl tcl-dev uuid-dev libcurl4-openssl-dev \
&& apt-get clean
CMD make
+29 -3
View File
@@ -1,9 +1,13 @@
![Current Release](https://img.shields.io/github/release/JohnSully/KeyDB.svg)
[![Build Status](https://travis-ci.org/JohnSully/KeyDB.svg?branch=unstable)](https://travis-ci.org/JohnSully/KeyDB) [![Join the chat at https://gitter.im/KeyDB/community](https://badges.gitter.im/KeyDB/community.svg)](https://gitter.im/KeyDB/community?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge)
![CI](https://github.com/JohnSully/KeyDB/workflows/CI/badge.svg?branch=unstable)
[![StackShare](http://img.shields.io/badge/tech-stack-0690fa.svg?style=flat)](https://stackshare.io/eq-alpha-technology-inc/eq-alpha-technology-inc)
##### New! Want to extend KeyDB with Javascript? Try [ModJS](https://github.com/JohnSully/ModJS)
##### Need Help? Check out our extensive [documentation](https://docs.keydb.dev).
##### Have feedback? Take our quick survey: https://www.surveymonkey.com/r/Y9XNS93
What is KeyDB?
--------------
@@ -86,13 +90,27 @@ Compiling is as simple as:
% make
You can enable flash support with:
To build with TLS support, you'll need OpenSSL development libraries (e.g.
libssl-dev on Debian/Ubuntu) and run:
% make BUILD_TLS=yes
To build with systemd support, you'll need systemd development libraries (such
as libsystemd-dev on Debian/Ubuntu or systemd-devel on CentOS) and run:
% make USE_SYSTEMD=yes
% make MALLOC=memkind
***Note that the following dependencies may be needed:
% sudo apt-get install autoconf autotools-dev libnuma-dev libtool
If TLS is built, running the tests with TLS enabled (you will need `tcl-tls`
installed):
% ./utils/gen-test-certs.sh
% ./runtest --tls
Fixing build problems with dependencies or cached build options
---------
@@ -175,6 +193,14 @@ as options using the command line. Examples:
All the options in keydb.conf are also supported as options using the command
line, with exactly the same name.
Running Redis with TLS:
------------------
Please consult the [TLS.md](TLS.md) file for more information on
how to use Redis with TLS.
Playing with KeyDB
------------------
+75
View File
@@ -0,0 +1,75 @@
TLS Support
===========
Getting Started
---------------
### Building
To build with TLS support you'll need OpenSSL development libraries (e.g.
libssl-dev on Debian/Ubuntu).
Run `make BUILD_TLS=yes`.
### Tests
To run Redis test suite with TLS, you'll need TLS support for TCL (i.e.
`tcl-tls` package on Debian/Ubuntu).
1. Run `./utils/gen-test-certs.sh` to generate a root CA and a server
certificate.
2. Run `./runtest --tls` or `./runtest-cluster --tls` to run Redis and Redis
Cluster tests in TLS mode.
### Running manually
To manually run a Redis server with TLS mode (assuming `gen-test-certs.sh` was
invoked so sample certificates/keys are available):
./src/redis-server --tls-port 6379 --port 0 \
--tls-cert-file ./tests/tls/redis.crt \
--tls-key-file ./tests/tls/redis.key \
--tls-ca-cert-file ./tests/tls/ca.crt
To connect to this Redis server with `redis-cli`:
./src/redis-cli --tls \
--cert ./tests/tls/redis.crt \
--key ./tests/tls/redis.key \
--cacert ./tests/tls/ca.crt
This will disable TCP and enable TLS on port 6379. It's also possible to have
both TCP and TLS available, but you'll need to assign different ports.
To make a Replica connect to the master using TLS, use `--tls-replication yes`,
and to make Redis Cluster use TLS across nodes use `--tls-cluster yes`.
Connections
-----------
All socket operations now go through a connection abstraction layer that hides
I/O and read/write event handling from the caller.
Note that unlike Redis, KeyDB fully supports multithreading of TLS connections.
To-Do List
----------
- [ ] redis-benchmark support. The current implementation is a mix of using
hiredis for parsing and basic networking (establishing connections), but
directly manipulating sockets for most actions. This will need to be cleaned
up for proper TLS support. The best approach is probably to migrate to hiredis
async mode.
- [ ] redis-cli `--slave` and `--rdb` support.
Multi-port
----------
Consider the implications of allowing TLS to be configured on a separate port,
making Redis listening on multiple ports:
1. Startup banner port notification
2. Proctitle
3. How slaves announce themselves
4. Cluster bus port calculation
+5 -1
View File
@@ -41,9 +41,13 @@ distclean:
.PHONY: distclean
ifeq ($(BUILD_TLS),yes)
HIREDIS_MAKE_FLAGS = USE_SSL=1
endif
hiredis: .make-prerequisites
@printf '%b %b\n' $(MAKECOLOR)MAKE$(ENDCOLOR) $(BINCOLOR)$@$(ENDCOLOR)
cd hiredis && $(MAKE) static
cd hiredis && $(MAKE) static $(HIREDIS_MAKE_FLAGS)
.PHONY: hiredis
+1 -1
View File
@@ -47,7 +47,7 @@ Hiredis
Hiredis uses the SDS string library, that must be the same version used inside Redis itself. Hiredis is also very critical for Sentinel. Historically Redis often used forked versions of hiredis in a way or the other. In order to upgrade it is advised to take a lot of care:
1. Check with diff if hiredis API changed and what impact it could have in Redis.
2. Make sure thet the SDS library inside Hiredis and inside Redis are compatible.
2. Make sure that the SDS library inside Hiredis and inside Redis are compatible.
3. After the upgrade, run the Redis Sentinel test.
4. Check manually that redis-cli and redis-benchmark behave as expecteed, since we have no tests for CLI utilities currently.
+1
View File
@@ -5,3 +5,4 @@
/*.dylib
/*.a
/*.pc
*.dSYM
+63 -11
View File
@@ -26,20 +26,72 @@ addons:
- libc6-dev-i386
- libc6-dbg:i386
- gcc-multilib
- g++-multilib
- valgrind
env:
- CFLAGS="-Werror"
- PRE="valgrind --track-origins=yes --leak-check=full"
- TARGET="32bit" TARGET_VARS="32bit-vars" CFLAGS="-Werror"
- TARGET="32bit" TARGET_VARS="32bit-vars" PRE="valgrind --track-origins=yes --leak-check=full"
- BITS="32"
- BITS="64"
script:
- EXTRA_CMAKE_OPTS="-DENABLE_EXAMPLES:BOOL=ON -DHIREDIS_SSL:BOOL=ON";
if [ "$TRAVIS_OS_NAME" == "osx" ]; then
if [ "$BITS" == "32" ]; then
CFLAGS="-m32 -Werror";
CXXFLAGS="-m32 -Werror";
LDFLAGS="-m32";
EXTRA_CMAKE_OPTS=;
else
CFLAGS="-Werror";
CXXFLAGS="-Werror";
fi;
else
TEST_PREFIX="valgrind --track-origins=yes --leak-check=full";
if [ "$BITS" == "32" ]; then
CFLAGS="-m32 -Werror";
CXXFLAGS="-m32 -Werror";
LDFLAGS="-m32";
EXTRA_CMAKE_OPTS=;
else
CFLAGS="-Werror";
CXXFLAGS="-Werror";
fi;
fi;
export CFLAGS CXXFLAGS LDFLAGS TEST_PREFIX EXTRA_CMAKE_OPTS
- mkdir build/ && cd build/
- cmake .. ${EXTRA_CMAKE_OPTS}
- make VERBOSE=1
- ctest -V
matrix:
exclude:
- os: osx
env: PRE="valgrind --track-origins=yes --leak-check=full"
include:
# Windows MinGW cross compile on Linux
- os: linux
dist: xenial
compiler: mingw
addons:
apt:
packages:
- ninja-build
- gcc-mingw-w64-x86-64
- g++-mingw-w64-x86-64
script:
- mkdir build && cd build
- CC=x86_64-w64-mingw32-gcc CXX=x86_64-w64-mingw32-g++ cmake .. -G Ninja -DCMAKE_BUILD_TYPE=Release -DCMAKE_BUILD_WITH_INSTALL_RPATH=on
- ninja -v
- os: osx
env: TARGET="32bit" TARGET_VARS="32bit-vars" PRE="valgrind --track-origins=yes --leak-check=full"
script: make $TARGET CFLAGS="$CFLAGS" && make check PRE="$PRE" && make $TARGET_VARS hiredis-example
# Windows MSVC 2017
- os: windows
compiler: msvc
env:
- MATRIX_EVAL="CC=cl.exe && CXX=cl.exe"
before_install:
- eval "${MATRIX_EVAL}"
install:
- choco install ninja
script:
- mkdir build && cd build
- cmd.exe /C '"C:\Program Files (x86)\Microsoft Visual Studio\2017\BuildTools\VC\Auxiliary\Build\vcvarsall.bat" amd64 &&
cmake .. -G Ninja -DCMAKE_BUILD_TYPE=Release &&
ninja -v'
- ctest -V
+13 -2
View File
@@ -12,6 +12,16 @@
compare to other values, casting might be necessary or can be removed, if
casting was applied before.
### 0.x.x (unreleased)
**BREAKING CHANGES**:
* Change `redisReply.len` to `size_t`, as it denotes the the size of a string
User code should compare this to `size_t` values as well.
If it was used to compare to other values, casting might be necessary or can be removed, if casting was applied before.
* `redisReplyObjectFunctions.createArray` now takes `size_t` for its length parameter.
### 0.14.0 (2018-09-25)
* Make string2ll static to fix conflict with Redis (Tom Lee [c3188b])
@@ -50,8 +60,9 @@
* Import latest upstream sds. This breaks applications that are linked against the old hiredis v0.13
* Fix warnings, when compiled with -Wshadow
* Make hiredis compile in Cygwin on Windows, now CI-tested
**BREAKING CHANGES**:
* Bulk and multi-bulk lengths less than -1 or greater than `LLONG_MAX` are now
protocol errors. This is consistent with the RESP specification. On 32-bit
platforms, the upper bound is lowered to `SIZE_MAX`.
* Remove backwards compatibility macro's
+90
View File
@@ -0,0 +1,90 @@
CMAKE_MINIMUM_REQUIRED(VERSION 3.4.0)
INCLUDE(GNUInstallDirs)
PROJECT(hiredis)
OPTION(ENABLE_SSL "Build hiredis_ssl for SSL support" OFF)
MACRO(getVersionBit name)
SET(VERSION_REGEX "^#define ${name} (.+)$")
FILE(STRINGS "${CMAKE_CURRENT_SOURCE_DIR}/hiredis.h"
VERSION_BIT REGEX ${VERSION_REGEX})
STRING(REGEX REPLACE ${VERSION_REGEX} "\\1" ${name} "${VERSION_BIT}")
ENDMACRO(getVersionBit)
getVersionBit(HIREDIS_MAJOR)
getVersionBit(HIREDIS_MINOR)
getVersionBit(HIREDIS_PATCH)
getVersionBit(HIREDIS_SONAME)
SET(VERSION "${HIREDIS_MAJOR}.${HIREDIS_MINOR}.${HIREDIS_PATCH}")
MESSAGE("Detected version: ${VERSION}")
PROJECT(hiredis VERSION "${VERSION}")
SET(ENABLE_EXAMPLES OFF CACHE BOOL "Enable building hiredis examples")
ADD_LIBRARY(hiredis SHARED
async.c
dict.c
hiredis.c
net.c
read.c
sds.c
sockcompat.c)
SET_TARGET_PROPERTIES(hiredis
PROPERTIES
VERSION "${HIREDIS_SONAME}")
IF(WIN32 OR MINGW)
TARGET_LINK_LIBRARIES(hiredis PRIVATE ws2_32)
ENDIF()
TARGET_INCLUDE_DIRECTORIES(hiredis PUBLIC .)
CONFIGURE_FILE(hiredis.pc.in hiredis.pc @ONLY)
INSTALL(TARGETS hiredis
DESTINATION "${CMAKE_INSTALL_LIBDIR}")
INSTALL(FILES hiredis.h read.h sds.h async.h
DESTINATION ${CMAKE_INSTALL_INCLUDEDIR}/hiredis)
INSTALL(DIRECTORY adapters
DESTINATION ${CMAKE_INSTALL_INCLUDEDIR}/hiredis)
INSTALL(FILES ${CMAKE_CURRENT_BINARY_DIR}/hiredis.pc
DESTINATION ${CMAKE_INSTALL_LIBDIR}/pkgconfig)
IF(ENABLE_SSL)
IF (NOT OPENSSL_ROOT_DIR)
IF (APPLE)
SET(OPENSSL_ROOT_DIR "/usr/local/opt/openssl")
ENDIF()
ENDIF()
FIND_PACKAGE(OpenSSL REQUIRED)
ADD_LIBRARY(hiredis_ssl SHARED
ssl.c)
TARGET_INCLUDE_DIRECTORIES(hiredis_ssl PRIVATE "${OPENSSL_INCLUDE_DIR}")
TARGET_LINK_LIBRARIES(hiredis_ssl PRIVATE ${OPENSSL_LIBRARIES})
CONFIGURE_FILE(hiredis_ssl.pc.in hiredis_ssl.pc @ONLY)
INSTALL(TARGETS hiredis_ssl
DESTINATION "${CMAKE_INSTALL_LIBDIR}")
INSTALL(FILES hiredis_ssl.h
DESTINATION ${CMAKE_INSTALL_INCLUDEDIR}/hiredis)
INSTALL(FILES ${CMAKE_CURRENT_BINARY_DIR}/hiredis_ssl.pc
DESTINATION ${CMAKE_INSTALL_LIBDIR}/pkgconfig)
ENDIF()
IF(NOT (WIN32 OR MINGW))
ENABLE_TESTING()
ADD_EXECUTABLE(hiredis-test test.c)
TARGET_LINK_LIBRARIES(hiredis-test hiredis)
ADD_TEST(NAME hiredis-test
COMMAND ${CMAKE_CURRENT_SOURCE_DIR}/test.sh)
ENDIF()
# Add examples
IF(ENABLE_EXAMPLES)
ADD_SUBDIRECTORY(examples)
ENDIF(ENABLE_EXAMPLES)
+83 -23
View File
@@ -3,11 +3,17 @@
# Copyright (C) 2010-2011 Pieter Noordhuis <pcnoordhuis at gmail dot com>
# This file is released under the BSD license, see the COPYING file
OBJ=net.o hiredis.o sds.o async.o read.o
OBJ=net.o hiredis.o sds.o async.o read.o sockcompat.o
SSL_OBJ=ssl.o
EXAMPLES=hiredis-example hiredis-example-libevent hiredis-example-libev hiredis-example-glib
ifeq ($(USE_SSL),1)
EXAMPLES+=hiredis-example-ssl hiredis-example-libevent-ssl
endif
TESTS=hiredis-test
LIBNAME=libhiredis
SSL_LIBNAME=libhiredis_ssl
PKGCONFNAME=hiredis.pc
SSL_PKGCONFNAME=hiredis_ssl.pc
HIREDIS_MAJOR=$(shell grep HIREDIS_MAJOR hiredis.h | awk '{print $$3}')
HIREDIS_MINOR=$(shell grep HIREDIS_MINOR hiredis.h | awk '{print $$3}')
@@ -39,7 +45,7 @@ export REDIS_TEST_CONFIG
CC:=$(shell sh -c 'type $${CC%% *} >/dev/null 2>/dev/null && echo $(CC) || echo gcc')
CXX:=$(shell sh -c 'type $${CXX%% *} >/dev/null 2>/dev/null && echo $(CXX) || echo g++')
OPTIMIZATION?=-O3
WARNINGS=-Wall -W -Wstrict-prototypes -Wwrite-strings
WARNINGS=-Wall -W -Wstrict-prototypes -Wwrite-strings -Wno-missing-field-initializers
DEBUG_FLAGS?= -g -ggdb
REAL_CFLAGS=$(OPTIMIZATION) -fPIC $(CPPFLAGS) $(CFLAGS) $(WARNINGS) $(DEBUG_FLAGS)
REAL_LDFLAGS=$(LDFLAGS)
@@ -49,12 +55,30 @@ STLIBSUFFIX=a
DYLIB_MINOR_NAME=$(LIBNAME).$(DYLIBSUFFIX).$(HIREDIS_SONAME)
DYLIB_MAJOR_NAME=$(LIBNAME).$(DYLIBSUFFIX).$(HIREDIS_MAJOR)
DYLIBNAME=$(LIBNAME).$(DYLIBSUFFIX)
DYLIB_MAKE_CMD=$(CC) -shared -Wl,-soname,$(DYLIB_MINOR_NAME) -o $(DYLIBNAME) $(LDFLAGS)
SSL_DYLIBNAME=$(SSL_LIBNAME).$(DYLIBSUFFIX)
DYLIB_MAKE_CMD=$(CC) -shared -Wl,-soname,$(DYLIB_MINOR_NAME)
STLIBNAME=$(LIBNAME).$(STLIBSUFFIX)
STLIB_MAKE_CMD=ar rcs $(STLIBNAME)
SSL_STLIBNAME=$(SSL_LIBNAME).$(STLIBSUFFIX)
STLIB_MAKE_CMD=$(AR) rcs
# Platform-specific overrides
uname_S := $(shell sh -c 'uname -s 2>/dev/null || echo not')
USE_SSL?=0
# This is required for test.c only
ifeq ($(USE_SSL),1)
CFLAGS+=-DHIREDIS_TEST_SSL
endif
ifeq ($(uname_S),Linux)
SSL_LDFLAGS=-lssl -lcrypto
else
OPENSSL_PREFIX?=/usr/local/opt/openssl
CFLAGS+=-I$(OPENSSL_PREFIX)/include
SSL_LDFLAGS+=-L$(OPENSSL_PREFIX)/lib -lssl -lcrypto
endif
ifeq ($(uname_S),SunOS)
REAL_LDFLAGS+= -ldl -lnsl -lsocket
DYLIB_MAKE_CMD=$(CC) -G -o $(DYLIBNAME) -h $(DYLIB_MINOR_NAME) $(LDFLAGS)
@@ -66,40 +90,61 @@ ifeq ($(uname_S),Darwin)
endif
all: $(DYLIBNAME) $(STLIBNAME) hiredis-test $(PKGCONFNAME)
ifeq ($(USE_SSL),1)
all: $(SSL_DYLIBNAME) $(SSL_STLIBNAME) $(SSL_PKGCONFNAME)
endif
# Deps (use make dep to generate this)
async.o: async.c fmacros.h async.h hiredis.h read.h sds.h net.h dict.c dict.h
dict.o: dict.c fmacros.h dict.h
hiredis.o: hiredis.c fmacros.h hiredis.h read.h sds.h net.h
net.o: net.c fmacros.h net.h hiredis.h read.h sds.h
hiredis.o: hiredis.c fmacros.h hiredis.h read.h sds.h net.h win32.h
net.o: net.c fmacros.h net.h hiredis.h read.h sds.h sockcompat.h win32.h
read.o: read.c fmacros.h read.h sds.h
sds.o: sds.c sds.h
sockcompat.o: sockcompat.c sockcompat.h
ssl.o: ssl.c hiredis.h
test.o: test.c fmacros.h hiredis.h read.h sds.h
$(DYLIBNAME): $(OBJ)
$(DYLIB_MAKE_CMD) $(OBJ)
$(DYLIB_MAKE_CMD) -o $(DYLIBNAME) $(OBJ) $(REAL_LDFLAGS)
$(STLIBNAME): $(OBJ)
$(STLIB_MAKE_CMD) $(OBJ)
$(STLIB_MAKE_CMD) $(STLIBNAME) $(OBJ)
$(SSL_DYLIBNAME): $(SSL_OBJ)
$(DYLIB_MAKE_CMD) -o $(SSL_DYLIBNAME) $(SSL_OBJ) $(REAL_LDFLAGS) $(SSL_LDFLAGS)
$(SSL_STLIBNAME): $(SSL_OBJ)
$(STLIB_MAKE_CMD) $(SSL_STLIBNAME) $(SSL_OBJ)
dynamic: $(DYLIBNAME)
static: $(STLIBNAME)
ifeq ($(USE_SSL),1)
dynamic: $(SSL_DYLIBNAME)
static: $(SSL_STLIBNAME)
endif
# Binaries:
hiredis-example-libevent: examples/example-libevent.c adapters/libevent.h $(STLIBNAME)
$(CC) -o examples/$@ $(REAL_CFLAGS) $(REAL_LDFLAGS) -I. $< -levent $(STLIBNAME)
$(CC) -o examples/$@ $(REAL_CFLAGS) -I. $< -levent $(STLIBNAME) $(REAL_LDFLAGS)
hiredis-example-libevent-ssl: examples/example-libevent-ssl.c adapters/libevent.h $(STLIBNAME) $(SSL_STLIBNAME)
$(CC) -o examples/$@ $(REAL_CFLAGS) -I. $< -levent $(STLIBNAME) $(SSL_STLIBNAME) $(REAL_LDFLAGS) $(SSL_LDFLAGS)
hiredis-example-libev: examples/example-libev.c adapters/libev.h $(STLIBNAME)
$(CC) -o examples/$@ $(REAL_CFLAGS) $(REAL_LDFLAGS) -I. $< -lev $(STLIBNAME)
$(CC) -o examples/$@ $(REAL_CFLAGS) -I. $< -lev $(STLIBNAME) $(REAL_LDFLAGS)
hiredis-example-glib: examples/example-glib.c adapters/glib.h $(STLIBNAME)
$(CC) -o examples/$@ $(REAL_CFLAGS) $(REAL_LDFLAGS) -I. $< $(shell pkg-config --cflags --libs glib-2.0) $(STLIBNAME)
$(CC) -o examples/$@ $(REAL_CFLAGS) -I. $< $(shell pkg-config --cflags --libs glib-2.0) $(STLIBNAME) $(REAL_LDFLAGS)
hiredis-example-ivykis: examples/example-ivykis.c adapters/ivykis.h $(STLIBNAME)
$(CC) -o examples/$@ $(REAL_CFLAGS) $(REAL_LDFLAGS) -I. $< -livykis $(STLIBNAME)
$(CC) -o examples/$@ $(REAL_CFLAGS) -I. $< -livykis $(STLIBNAME) $(REAL_LDFLAGS)
hiredis-example-macosx: examples/example-macosx.c adapters/macosx.h $(STLIBNAME)
$(CC) -o examples/$@ $(REAL_CFLAGS) $(REAL_LDFLAGS) -I. $< -framework CoreFoundation $(STLIBNAME)
$(CC) -o examples/$@ $(REAL_CFLAGS) -I. $< -framework CoreFoundation $(STLIBNAME) $(REAL_LDFLAGS)
hiredis-example-ssl: examples/example-ssl.c $(STLIBNAME) $(SSL_STLIBNAME)
$(CC) -o examples/$@ $(REAL_CFLAGS) -I. $< $(STLIBNAME) $(SSL_STLIBNAME) $(REAL_LDFLAGS) $(SSL_LDFLAGS)
ifndef AE_DIR
hiredis-example-ae:
@@ -116,7 +161,7 @@ hiredis-example-libuv:
@false
else
hiredis-example-libuv: examples/example-libuv.c adapters/libuv.h $(STLIBNAME)
$(CC) -o examples/$@ $(REAL_CFLAGS) $(REAL_LDFLAGS) -I. -I$(LIBUV_DIR)/include $< $(LIBUV_DIR)/.libs/libuv.a -lpthread -lrt $(STLIBNAME)
$(CC) -o examples/$@ $(REAL_CFLAGS) -I. -I$(LIBUV_DIR)/include $< $(LIBUV_DIR)/.libs/libuv.a -lpthread -lrt $(STLIBNAME) $(REAL_LDFLAGS)
endif
ifeq ($(and $(QT_MOC),$(QT_INCLUDE_DIR),$(QT_LIBRARY_DIR)),)
@@ -133,32 +178,33 @@ hiredis-example-qt: examples/example-qt.cpp adapters/qt.h $(STLIBNAME)
endif
hiredis-example: examples/example.c $(STLIBNAME)
$(CC) -o examples/$@ $(REAL_CFLAGS) $(REAL_LDFLAGS) -I. $< $(STLIBNAME)
$(CC) -o examples/$@ $(REAL_CFLAGS) -I. $< $(STLIBNAME) $(REAL_LDFLAGS)
examples: $(EXAMPLES)
hiredis-test: test.o $(STLIBNAME)
TEST_LIBS = $(STLIBNAME)
ifeq ($(USE_SSL),1)
TEST_LIBS += $(SSL_STLIBNAME) -lssl -lcrypto -lpthread
endif
hiredis-test: test.o $(TEST_LIBS)
hiredis-%: %.o $(STLIBNAME)
$(CC) $(REAL_CFLAGS) -o $@ $(REAL_LDFLAGS) $< $(STLIBNAME)
$(CC) $(REAL_CFLAGS) -o $@ $< $(TEST_LIBS) $(REAL_LDFLAGS)
test: hiredis-test
./hiredis-test
check: hiredis-test
@echo "$$REDIS_TEST_CONFIG" | $(REDIS_SERVER) -
$(PRE) ./hiredis-test -h 127.0.0.1 -p $(REDIS_PORT) -s /tmp/hiredis-test-redis.sock || \
( kill `cat /tmp/hiredis-test-redis.pid` && false )
kill `cat /tmp/hiredis-test-redis.pid`
TEST_SSL=$(USE_SSL) ./test.sh
.c.o:
$(CC) -std=c99 -pedantic -c $(REAL_CFLAGS) $<
clean:
rm -rf $(DYLIBNAME) $(STLIBNAME) $(TESTS) $(PKGCONFNAME) examples/hiredis-example* *.o *.gcda *.gcno *.gcov
rm -rf $(DYLIBNAME) $(STLIBNAME) $(SSL_DYLIBNAME) $(SSL_STLIBNAME) $(TESTS) $(PKGCONFNAME) examples/hiredis-example* *.o *.gcda *.gcno *.gcov
dep:
$(CC) -MM *.c
$(CC) $(CPPFLAGS) $(CFLAGS) -MM *.c
INSTALL?= cp -pPR
@@ -175,6 +221,20 @@ $(PKGCONFNAME): hiredis.h
@echo Libs: -L\$${libdir} -lhiredis >> $@
@echo Cflags: -I\$${includedir} -D_FILE_OFFSET_BITS=64 >> $@
$(SSL_PKGCONFNAME): hiredis.h
@echo "Generating $@ for pkgconfig..."
@echo prefix=$(PREFIX) > $@
@echo exec_prefix=\$${prefix} >> $@
@echo libdir=$(PREFIX)/$(LIBRARY_PATH) >> $@
@echo includedir=$(PREFIX)/$(INCLUDE_PATH) >> $@
@echo >> $@
@echo Name: hiredis_ssl >> $@
@echo Description: SSL Support for hiredis. >> $@
@echo Version: $(HIREDIS_MAJOR).$(HIREDIS_MINOR).$(HIREDIS_PATCH) >> $@
@echo Requires: hiredis >> $@
@echo Libs: -L\$${libdir} -lhiredis_ssl >> $@
@echo Libs.private: -lssl -lcrypto >> $@
install: $(DYLIBNAME) $(STLIBNAME) $(PKGCONFNAME)
mkdir -p $(INSTALL_INCLUDE_PATH) $(INSTALL_INCLUDE_PATH)/adapters $(INSTALL_LIBRARY_PATH)
$(INSTALL) hiredis.h async.h read.h sds.h $(INSTALL_INCLUDE_PATH)
+2 -1
View File
@@ -286,6 +286,7 @@ return `REDIS_ERR`. The function to set the disconnect callback has the followin
```c
int redisAsyncSetDisconnectCallback(redisAsyncContext *ac, redisDisconnectCallback *fn);
```
`ac->data` may be used to pass user data to this callback, the same can be done for redisConnectCallback.
### Sending commands and their callbacks
In an asynchronous context, commands are automatically pipelined due to the nature of an event loop.
@@ -406,6 +407,6 @@ as soon as possible in order to prevent allocation of useless memory.
## AUTHORS
Hiredis was written by Salvatore Sanfilippo (antirez at gmail) and
Pieter Noordhuis (pcnoordhuis at gmail) and is released under the BSD license.
Pieter Noordhuis (pcnoordhuis at gmail) and is released under the BSD license.
Hiredis is currently maintained by Matt Stancliff (matt at genges dot com) and
Jan-Erik Rediger (janerik at fnordig dot com)
+88 -24
View File
@@ -34,48 +34,113 @@
#include "../hiredis.h"
#include "../async.h"
#define REDIS_LIBEVENT_DELETED 0x01
#define REDIS_LIBEVENT_ENTERED 0x02
typedef struct redisLibeventEvents {
redisAsyncContext *context;
struct event *rev, *wev;
struct event *ev;
struct event_base *base;
struct timeval tv;
short flags;
short state;
} redisLibeventEvents;
static void redisLibeventReadEvent(int fd, short event, void *arg) {
((void)fd); ((void)event);
redisLibeventEvents *e = (redisLibeventEvents*)arg;
redisAsyncHandleRead(e->context);
static void redisLibeventDestroy(redisLibeventEvents *e) {
free(e);
}
static void redisLibeventWriteEvent(int fd, short event, void *arg) {
((void)fd); ((void)event);
static void redisLibeventHandler(int fd, short event, void *arg) {
((void)fd);
redisLibeventEvents *e = (redisLibeventEvents*)arg;
redisAsyncHandleWrite(e->context);
e->state |= REDIS_LIBEVENT_ENTERED;
#define CHECK_DELETED() if (e->state & REDIS_LIBEVENT_DELETED) {\
redisLibeventDestroy(e);\
return; \
}
if ((event & EV_TIMEOUT) && (e->state & REDIS_LIBEVENT_DELETED) == 0) {
redisAsyncHandleTimeout(e->context);
CHECK_DELETED();
}
if ((event & EV_READ) && e->context && (e->state & REDIS_LIBEVENT_DELETED) == 0) {
redisAsyncHandleRead(e->context);
CHECK_DELETED();
}
if ((event & EV_WRITE) && e->context && (e->state & REDIS_LIBEVENT_DELETED) == 0) {
redisAsyncHandleWrite(e->context);
CHECK_DELETED();
}
e->state &= ~REDIS_LIBEVENT_ENTERED;
#undef CHECK_DELETED
}
static void redisLibeventUpdate(void *privdata, short flag, int isRemove) {
redisLibeventEvents *e = (redisLibeventEvents *)privdata;
const struct timeval *tv = e->tv.tv_sec || e->tv.tv_usec ? &e->tv : NULL;
if (isRemove) {
if ((e->flags & flag) == 0) {
return;
} else {
e->flags &= ~flag;
}
} else {
if (e->flags & flag) {
return;
} else {
e->flags |= flag;
}
}
event_del(e->ev);
event_assign(e->ev, e->base, e->context->c.fd, e->flags | EV_PERSIST,
redisLibeventHandler, privdata);
event_add(e->ev, tv);
}
static void redisLibeventAddRead(void *privdata) {
redisLibeventEvents *e = (redisLibeventEvents*)privdata;
event_add(e->rev,NULL);
redisLibeventUpdate(privdata, EV_READ, 0);
}
static void redisLibeventDelRead(void *privdata) {
redisLibeventEvents *e = (redisLibeventEvents*)privdata;
event_del(e->rev);
redisLibeventUpdate(privdata, EV_READ, 1);
}
static void redisLibeventAddWrite(void *privdata) {
redisLibeventEvents *e = (redisLibeventEvents*)privdata;
event_add(e->wev,NULL);
redisLibeventUpdate(privdata, EV_WRITE, 0);
}
static void redisLibeventDelWrite(void *privdata) {
redisLibeventEvents *e = (redisLibeventEvents*)privdata;
event_del(e->wev);
redisLibeventUpdate(privdata, EV_WRITE, 1);
}
static void redisLibeventCleanup(void *privdata) {
redisLibeventEvents *e = (redisLibeventEvents*)privdata;
event_free(e->rev);
event_free(e->wev);
free(e);
if (!e) {
return;
}
event_del(e->ev);
event_free(e->ev);
e->ev = NULL;
if (e->state & REDIS_LIBEVENT_ENTERED) {
e->state |= REDIS_LIBEVENT_DELETED;
} else {
redisLibeventDestroy(e);
}
}
static void redisLibeventSetTimeout(void *privdata, struct timeval tv) {
redisLibeventEvents *e = (redisLibeventEvents *)privdata;
short flags = e->flags;
e->flags = 0;
e->tv = tv;
redisLibeventUpdate(e, flags, 0);
}
static int redisLibeventAttach(redisAsyncContext *ac, struct event_base *base) {
@@ -87,7 +152,7 @@ static int redisLibeventAttach(redisAsyncContext *ac, struct event_base *base) {
return REDIS_ERR;
/* Create container for context and r/w events */
e = (redisLibeventEvents*)malloc(sizeof(*e));
e = (redisLibeventEvents*)calloc(1, sizeof(*e));
e->context = ac;
/* Register functions to start/stop listening for events */
@@ -96,13 +161,12 @@ static int redisLibeventAttach(redisAsyncContext *ac, struct event_base *base) {
ac->ev.addWrite = redisLibeventAddWrite;
ac->ev.delWrite = redisLibeventDelWrite;
ac->ev.cleanup = redisLibeventCleanup;
ac->ev.scheduleTimer = redisLibeventSetTimeout;
ac->ev.data = e;
/* Initialize and install read/write events */
e->rev = event_new(base, c->fd, EV_READ, redisLibeventReadEvent, e);
e->wev = event_new(base, c->fd, EV_WRITE, redisLibeventWriteEvent, e);
event_add(e->rev, NULL);
event_add(e->wev, NULL);
e->ev = event_new(base, c->fd, EV_READ | EV_WRITE, redisLibeventHandler, e);
e->base = base;
return REDIS_OK;
}
#endif
+4 -3
View File
@@ -5,8 +5,9 @@ environment:
CC: gcc
- CYG_BASH: C:\cygwin\bin\bash
CC: gcc
TARGET: 32bit
TARGET_VARS: 32bit-vars
CFLAGS: -m32
CXXFLAGS: -m32
LDFLAGS: -m32
clone_depth: 1
@@ -20,4 +21,4 @@ install:
build_script:
- 'echo building...'
- '%CYG_BASH% -lc "cd $APPVEYOR_BUILD_FOLDER; exec 0</dev/null; make LDFLAGS=$LDFLAGS CC=$CC $TARGET CFLAGS=$CFLAGS && make LDFLAGS=$LDFLAGS CC=$CC $TARGET_VARS hiredis-example"'
- '%CYG_BASH% -lc "cd $APPVEYOR_BUILD_FOLDER; exec 0</dev/null; mkdir build && cd build && cmake .. -G \"Unix Makefiles\" && make VERBOSE=1"'
+113 -63
View File
@@ -32,7 +32,9 @@
#include "fmacros.h"
#include <stdlib.h>
#include <string.h>
#ifndef _MSC_VER
#include <strings.h>
#endif
#include <assert.h>
#include <ctype.h>
#include <errno.h>
@@ -40,22 +42,9 @@
#include "net.h"
#include "dict.c"
#include "sds.h"
#include "win32.h"
#define _EL_ADD_READ(ctx) do { \
if ((ctx)->ev.addRead) (ctx)->ev.addRead((ctx)->ev.data); \
} while(0)
#define _EL_DEL_READ(ctx) do { \
if ((ctx)->ev.delRead) (ctx)->ev.delRead((ctx)->ev.data); \
} while(0)
#define _EL_ADD_WRITE(ctx) do { \
if ((ctx)->ev.addWrite) (ctx)->ev.addWrite((ctx)->ev.data); \
} while(0)
#define _EL_DEL_WRITE(ctx) do { \
if ((ctx)->ev.delWrite) (ctx)->ev.delWrite((ctx)->ev.data); \
} while(0)
#define _EL_CLEANUP(ctx) do { \
if ((ctx)->ev.cleanup) (ctx)->ev.cleanup((ctx)->ev.data); \
} while(0);
#include "async_private.h"
/* Forward declaration of function in hiredis.c */
int __redisAppendCommand(redisContext *c, const char *cmd, size_t len);
@@ -126,6 +115,7 @@ static redisAsyncContext *redisAsyncInitialize(redisContext *c) {
ac->ev.addWrite = NULL;
ac->ev.delWrite = NULL;
ac->ev.cleanup = NULL;
ac->ev.scheduleTimer = NULL;
ac->onConnect = NULL;
ac->onDisconnect = NULL;
@@ -150,56 +140,52 @@ static void __redisAsyncCopyError(redisAsyncContext *ac) {
ac->errstr = c->errstr;
}
redisAsyncContext *redisAsyncConnect(const char *ip, int port) {
redisAsyncContext *redisAsyncConnectWithOptions(const redisOptions *options) {
redisOptions myOptions = *options;
redisContext *c;
redisAsyncContext *ac;
c = redisConnectNonBlock(ip,port);
if (c == NULL)
myOptions.options |= REDIS_OPT_NONBLOCK;
c = redisConnectWithOptions(&myOptions);
if (c == NULL) {
return NULL;
}
ac = redisAsyncInitialize(c);
if (ac == NULL) {
redisFree(c);
return NULL;
}
__redisAsyncCopyError(ac);
return ac;
}
redisAsyncContext *redisAsyncConnect(const char *ip, int port) {
redisOptions options = {0};
REDIS_OPTIONS_SET_TCP(&options, ip, port);
return redisAsyncConnectWithOptions(&options);
}
redisAsyncContext *redisAsyncConnectBind(const char *ip, int port,
const char *source_addr) {
redisContext *c = redisConnectBindNonBlock(ip,port,source_addr);
redisAsyncContext *ac = redisAsyncInitialize(c);
__redisAsyncCopyError(ac);
return ac;
redisOptions options = {0};
REDIS_OPTIONS_SET_TCP(&options, ip, port);
options.endpoint.tcp.source_addr = source_addr;
return redisAsyncConnectWithOptions(&options);
}
redisAsyncContext *redisAsyncConnectBindWithReuse(const char *ip, int port,
const char *source_addr) {
redisContext *c = redisConnectBindNonBlockWithReuse(ip,port,source_addr);
redisAsyncContext *ac = redisAsyncInitialize(c);
__redisAsyncCopyError(ac);
return ac;
redisOptions options = {0};
REDIS_OPTIONS_SET_TCP(&options, ip, port);
options.options |= REDIS_OPT_REUSEADDR;
options.endpoint.tcp.source_addr = source_addr;
return redisAsyncConnectWithOptions(&options);
}
redisAsyncContext *redisAsyncConnectUnix(const char *path) {
redisContext *c;
redisAsyncContext *ac;
c = redisConnectUnixNonBlock(path);
if (c == NULL)
return NULL;
ac = redisAsyncInitialize(c);
if (ac == NULL) {
redisFree(c);
return NULL;
}
__redisAsyncCopyError(ac);
return ac;
redisOptions options = {0};
REDIS_OPTIONS_SET_UNIX(&options, path);
return redisAsyncConnectWithOptions(&options);
}
int redisAsyncSetConnectCallback(redisAsyncContext *ac, redisConnectCallback *fn) {
@@ -328,7 +314,7 @@ void redisAsyncFree(redisAsyncContext *ac) {
}
/* Helper function to make the disconnect happen and clean up. */
static void __redisAsyncDisconnect(redisAsyncContext *ac) {
void __redisAsyncDisconnect(redisAsyncContext *ac) {
redisContext *c = &(ac->c);
/* Make sure error is accessible if there is any */
@@ -344,9 +330,15 @@ static void __redisAsyncDisconnect(redisAsyncContext *ac) {
c->flags |= REDIS_DISCONNECTING;
}
/* cleanup event library on disconnect.
* this is safe to call multiple times */
_EL_CLEANUP(ac);
/* For non-clean disconnects, __redisAsyncFree() will execute pending
* callbacks with a NULL-reply. */
__redisAsyncFree(ac);
if (!(c->flags & REDIS_NO_AUTO_FREE)) {
__redisAsyncFree(ac);
}
}
/* Tries to do a clean disconnect from Redis, meaning it stops new commands
@@ -358,6 +350,9 @@ static void __redisAsyncDisconnect(redisAsyncContext *ac) {
void redisAsyncDisconnect(redisAsyncContext *ac) {
redisContext *c = &(ac->c);
c->flags |= REDIS_DISCONNECTING;
/** unset the auto-free flag here, because disconnect undoes this */
c->flags &= ~REDIS_NO_AUTO_FREE;
if (!(c->flags & REDIS_IN_CALLBACK) && ac->replies.head == NULL)
__redisAsyncDisconnect(ac);
}
@@ -408,7 +403,7 @@ static int __redisGetSubscribeCallback(redisAsyncContext *ac, redisReply *reply,
assert(reply->element[2]->type == REDIS_REPLY_INTEGER);
/* Unset subscribed flag only when no pipelined pending subscribe. */
if (reply->element[2]->integer == 0
if (reply->element[2]->integer == 0
&& dictSize(ac->sub.channels) == 0
&& dictSize(ac->sub.patterns) == 0)
c->flags &= ~REDIS_SUBSCRIBED;
@@ -524,6 +519,18 @@ static int __redisAsyncHandleConnect(redisAsyncContext *ac) {
}
}
void redisAsyncRead(redisAsyncContext *ac) {
redisContext *c = &(ac->c);
if (redisBufferRead(c) == REDIS_ERR) {
__redisAsyncDisconnect(ac);
} else {
/* Always re-schedule reads */
_EL_ADD_READ(ac);
redisProcessCallbacks(ac);
}
}
/* This function should be called when the socket is readable.
* It processes all replies that can be read and executes their callbacks.
*/
@@ -539,28 +546,13 @@ void redisAsyncHandleRead(redisAsyncContext *ac) {
return;
}
if (redisBufferRead(c) == REDIS_ERR) {
__redisAsyncDisconnect(ac);
} else {
/* Always re-schedule reads */
_EL_ADD_READ(ac);
redisProcessCallbacks(ac);
}
c->funcs->async_read(ac);
}
void redisAsyncHandleWrite(redisAsyncContext *ac) {
void redisAsyncWrite(redisAsyncContext *ac) {
redisContext *c = &(ac->c);
int done = 0;
if (!(c->flags & REDIS_CONNECTED)) {
/* Abort connect was not successful. */
if (__redisAsyncHandleConnect(ac) != REDIS_OK)
return;
/* Try again later when the context is still not connected. */
if (!(c->flags & REDIS_CONNECTED))
return;
}
if (redisBufferWrite(c,&done) == REDIS_ERR) {
__redisAsyncDisconnect(ac);
} else {
@@ -575,6 +567,51 @@ void redisAsyncHandleWrite(redisAsyncContext *ac) {
}
}
void redisAsyncHandleWrite(redisAsyncContext *ac) {
redisContext *c = &(ac->c);
if (!(c->flags & REDIS_CONNECTED)) {
/* Abort connect was not successful. */
if (__redisAsyncHandleConnect(ac) != REDIS_OK)
return;
/* Try again later when the context is still not connected. */
if (!(c->flags & REDIS_CONNECTED))
return;
}
c->funcs->async_write(ac);
}
void __redisSetError(redisContext *c, int type, const char *str);
void redisAsyncHandleTimeout(redisAsyncContext *ac) {
redisContext *c = &(ac->c);
redisCallback cb;
if ((c->flags & REDIS_CONNECTED) && ac->replies.head == NULL) {
/* Nothing to do - just an idle timeout */
return;
}
if (!c->err) {
__redisSetError(c, REDIS_ERR_TIMEOUT, "Timeout");
}
if (!(c->flags & REDIS_CONNECTED) && ac->onConnect) {
ac->onConnect(ac, REDIS_ERR);
}
while (__redisShiftCallback(&ac->replies, &cb) == REDIS_OK) {
__redisRunCallback(ac, &cb, NULL);
}
/**
* TODO: Don't automatically sever the connection,
* rather, allow to ignore <x> responses before the queue is clear
*/
__redisAsyncDisconnect(ac);
}
/* Sets a pointer to the first argument and its length starting at p. Returns
* the number of bytes to skip to get to the following argument. */
static const char *nextArgument(const char *start, const char **str, size_t *len) {
@@ -714,3 +751,16 @@ int redisAsyncFormattedCommand(redisAsyncContext *ac, redisCallbackFn *fn, void
int status = __redisAsyncCommand(ac,fn,privdata,cmd,len);
return status;
}
void redisAsyncSetTimeout(redisAsyncContext *ac, struct timeval tv) {
if (!ac->c.timeout) {
ac->c.timeout = calloc(1, sizeof(tv));
}
if (tv.tv_sec == ac->c.timeout->tv_sec &&
tv.tv_usec == ac->c.timeout->tv_usec) {
return;
}
*ac->c.timeout = tv;
}
+8
View File
@@ -57,6 +57,7 @@ typedef struct redisCallbackList {
/* Connection callback prototypes */
typedef void (redisDisconnectCallback)(const struct redisAsyncContext*, int status);
typedef void (redisConnectCallback)(const struct redisAsyncContext*, int status);
typedef void(redisTimerCallback)(void *timer, void *privdata);
/* Context for an async connection to Redis */
typedef struct redisAsyncContext {
@@ -81,6 +82,7 @@ typedef struct redisAsyncContext {
void (*addWrite)(void *privdata);
void (*delWrite)(void *privdata);
void (*cleanup)(void *privdata);
void (*scheduleTimer)(void *privdata, struct timeval tv);
} ev;
/* Called when either the connection is terminated due to an error or per
@@ -106,6 +108,7 @@ typedef struct redisAsyncContext {
} redisAsyncContext;
/* Functions that proxy to hiredis */
redisAsyncContext *redisAsyncConnectWithOptions(const redisOptions *options);
redisAsyncContext *redisAsyncConnect(const char *ip, int port);
redisAsyncContext *redisAsyncConnectBind(const char *ip, int port, const char *source_addr);
redisAsyncContext *redisAsyncConnectBindWithReuse(const char *ip, int port,
@@ -113,12 +116,17 @@ redisAsyncContext *redisAsyncConnectBindWithReuse(const char *ip, int port,
redisAsyncContext *redisAsyncConnectUnix(const char *path);
int redisAsyncSetConnectCallback(redisAsyncContext *ac, redisConnectCallback *fn);
int redisAsyncSetDisconnectCallback(redisAsyncContext *ac, redisDisconnectCallback *fn);
void redisAsyncSetTimeout(redisAsyncContext *ac, struct timeval tv);
void redisAsyncDisconnect(redisAsyncContext *ac);
void redisAsyncFree(redisAsyncContext *ac);
/* Handle read/write events */
void redisAsyncHandleRead(redisAsyncContext *ac);
void redisAsyncHandleWrite(redisAsyncContext *ac);
void redisAsyncHandleTimeout(redisAsyncContext *ac);
void redisAsyncRead(redisAsyncContext *ac);
void redisAsyncWrite(redisAsyncContext *ac);
/* Command functions for an async context. Write the command to the
* output buffer and register the provided callback. */
+72
View File
@@ -0,0 +1,72 @@
/*
* Copyright (c) 2009-2011, Salvatore Sanfilippo <antirez at gmail dot com>
* Copyright (c) 2010-2011, Pieter Noordhuis <pcnoordhuis at gmail dot com>
*
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions are met:
*
* * Redistributions of source code must retain the above copyright notice,
* this list of conditions and the following disclaimer.
* * Redistributions in binary form must reproduce the above copyright
* notice, this list of conditions and the following disclaimer in the
* documentation and/or other materials provided with the distribution.
* * Neither the name of Redis nor the names of its contributors may be used
* to endorse or promote products derived from this software without
* specific prior written permission.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
* AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
* ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
* LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
* CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
* SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
* INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
* CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
* ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
* POSSIBILITY OF SUCH DAMAGE.
*/
#ifndef __HIREDIS_ASYNC_PRIVATE_H
#define __HIREDIS_ASYNC_PRIVATE_H
#define _EL_ADD_READ(ctx) \
do { \
refreshTimeout(ctx); \
if ((ctx)->ev.addRead) (ctx)->ev.addRead((ctx)->ev.data); \
} while (0)
#define _EL_DEL_READ(ctx) do { \
if ((ctx)->ev.delRead) (ctx)->ev.delRead((ctx)->ev.data); \
} while(0)
#define _EL_ADD_WRITE(ctx) \
do { \
refreshTimeout(ctx); \
if ((ctx)->ev.addWrite) (ctx)->ev.addWrite((ctx)->ev.data); \
} while (0)
#define _EL_DEL_WRITE(ctx) do { \
if ((ctx)->ev.delWrite) (ctx)->ev.delWrite((ctx)->ev.data); \
} while(0)
#define _EL_CLEANUP(ctx) do { \
if ((ctx)->ev.cleanup) (ctx)->ev.cleanup((ctx)->ev.data); \
ctx->ev.cleanup = NULL; \
} while(0);
static inline void refreshTimeout(redisAsyncContext *ctx) {
if (ctx->c.timeout && ctx->ev.scheduleTimer &&
(ctx->c.timeout->tv_sec || ctx->c.timeout->tv_usec)) {
ctx->ev.scheduleTimer(ctx->ev.data, *ctx->c.timeout);
// } else {
// printf("Not scheduling timer.. (tmo=%p)\n", ctx->c.timeout);
// if (ctx->c.timeout){
// printf("tv_sec: %u. tv_usec: %u\n", ctx->c.timeout->tv_sec,
// ctx->c.timeout->tv_usec);
// }
}
}
void __redisAsyncDisconnect(redisAsyncContext *ac);
void redisProcessCallbacks(redisAsyncContext *ac);
#endif /* __HIREDIS_ASYNC_PRIVATE_H */
+46
View File
@@ -0,0 +1,46 @@
INCLUDE(FindPkgConfig)
# Check for GLib
PKG_CHECK_MODULES(GLIB2 glib-2.0)
if (GLIB2_FOUND)
INCLUDE_DIRECTORIES(${GLIB2_INCLUDE_DIRS})
LINK_DIRECTORIES(${GLIB2_LIBRARY_DIRS})
ADD_EXECUTABLE(example-glib example-glib.c)
TARGET_LINK_LIBRARIES(example-glib hiredis ${GLIB2_LIBRARIES})
ENDIF(GLIB2_FOUND)
FIND_PATH(LIBEV ev.h
HINTS /usr/local /usr/opt/local
ENV LIBEV_INCLUDE_DIR)
if (LIBEV)
# Just compile and link with libev
ADD_EXECUTABLE(example-libev example-libev.c)
TARGET_LINK_LIBRARIES(example-libev hiredis ev)
ENDIF()
FIND_PATH(LIBEVENT event.h)
if (LIBEVENT)
ADD_EXECUTABLE(example-libevent example-libevent)
TARGET_LINK_LIBRARIES(example-libevent hiredis event)
ENDIF()
FIND_PATH(LIBUV uv.h)
IF (LIBUV)
ADD_EXECUTABLE(example-libuv example-libuv.c)
TARGET_LINK_LIBRARIES(example-libuv hiredis uv)
ENDIF()
IF (APPLE)
FIND_LIBRARY(CF CoreFoundation)
ADD_EXECUTABLE(example-macosx example-macosx.c)
TARGET_LINK_LIBRARIES(example-macosx hiredis ${CF})
ENDIF()
IF (ENABLE_SSL)
ADD_EXECUTABLE(example-ssl example-ssl.c)
TARGET_LINK_LIBRARIES(example-ssl hiredis hiredis_ssl)
ENDIF()
ADD_EXECUTABLE(example example.c)
TARGET_LINK_LIBRARIES(example hiredis)
+73
View File
@@ -0,0 +1,73 @@
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <signal.h>
#include <hiredis.h>
#include <hiredis_ssl.h>
#include <async.h>
#include <adapters/libevent.h>
void getCallback(redisAsyncContext *c, void *r, void *privdata) {
redisReply *reply = r;
if (reply == NULL) return;
printf("argv[%s]: %s\n", (char*)privdata, reply->str);
/* Disconnect after receiving the reply to GET */
redisAsyncDisconnect(c);
}
void connectCallback(const redisAsyncContext *c, int status) {
if (status != REDIS_OK) {
printf("Error: %s\n", c->errstr);
return;
}
printf("Connected...\n");
}
void disconnectCallback(const redisAsyncContext *c, int status) {
if (status != REDIS_OK) {
printf("Error: %s\n", c->errstr);
return;
}
printf("Disconnected...\n");
}
int main (int argc, char **argv) {
signal(SIGPIPE, SIG_IGN);
struct event_base *base = event_base_new();
if (argc < 5) {
fprintf(stderr,
"Usage: %s <key> <host> <port> <cert> <certKey> [ca]\n", argv[0]);
exit(1);
}
const char *value = argv[1];
size_t nvalue = strlen(value);
const char *hostname = argv[2];
int port = atoi(argv[3]);
const char *cert = argv[4];
const char *certKey = argv[5];
const char *caCert = argc > 5 ? argv[6] : NULL;
redisAsyncContext *c = redisAsyncConnect(hostname, port);
if (c->err) {
/* Let *c leak for now... */
printf("Error: %s\n", c->errstr);
return 1;
}
if (redisSecureConnection(&c->c, caCert, cert, certKey, "sni") != REDIS_OK) {
printf("SSL Error!\n");
exit(1);
}
redisLibeventAttach(c,base);
redisAsyncSetConnectCallback(c,connectCallback);
redisAsyncSetDisconnectCallback(c,disconnectCallback);
redisAsyncCommand(c, NULL, NULL, "SET key %b", value, nvalue);
redisAsyncCommand(c, getCallback, (char*)"end-1", "GET key");
event_base_dispatch(base);
return 0;
}
+13 -2
View File
@@ -9,7 +9,12 @@
void getCallback(redisAsyncContext *c, void *r, void *privdata) {
redisReply *reply = r;
if (reply == NULL) return;
if (reply == NULL) {
if (c->errstr) {
printf("errstr: %s\n", c->errstr);
}
return;
}
printf("argv[%s]: %s\n", (char*)privdata, reply->str);
/* Disconnect after receiving the reply to GET */
@@ -35,8 +40,14 @@ void disconnectCallback(const redisAsyncContext *c, int status) {
int main (int argc, char **argv) {
signal(SIGPIPE, SIG_IGN);
struct event_base *base = event_base_new();
redisOptions options = {0};
REDIS_OPTIONS_SET_TCP(&options, "127.0.0.1", 6379);
struct timeval tv = {0};
tv.tv_sec = 1;
options.timeout = &tv;
redisAsyncContext *c = redisAsyncConnect("127.0.0.1", 6379);
redisAsyncContext *c = redisAsyncConnectWithOptions(&options);
if (c->err) {
/* Let *c leak for now... */
printf("Error: %s\n", c->errstr);
+97
View File
@@ -0,0 +1,97 @@
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <hiredis.h>
#include <hiredis_ssl.h>
int main(int argc, char **argv) {
unsigned int j;
redisContext *c;
redisReply *reply;
if (argc < 4) {
printf("Usage: %s <host> <port> <cert> <key> [ca]\n", argv[0]);
exit(1);
}
const char *hostname = (argc > 1) ? argv[1] : "127.0.0.1";
int port = atoi(argv[2]);
const char *cert = argv[3];
const char *key = argv[4];
const char *ca = argc > 4 ? argv[5] : NULL;
struct timeval tv = { 1, 500000 }; // 1.5 seconds
redisOptions options = {0};
REDIS_OPTIONS_SET_TCP(&options, hostname, port);
options.timeout = &tv;
c = redisConnectWithOptions(&options);
if (c == NULL || c->err) {
if (c) {
printf("Connection error: %s\n", c->errstr);
redisFree(c);
} else {
printf("Connection error: can't allocate redis context\n");
}
exit(1);
}
if (redisSecureConnection(c, ca, cert, key, "sni") != REDIS_OK) {
printf("Couldn't initialize SSL!\n");
printf("Error: %s\n", c->errstr);
redisFree(c);
exit(1);
}
/* PING server */
reply = redisCommand(c,"PING");
printf("PING: %s\n", reply->str);
freeReplyObject(reply);
/* Set a key */
reply = redisCommand(c,"SET %s %s", "foo", "hello world");
printf("SET: %s\n", reply->str);
freeReplyObject(reply);
/* Set a key using binary safe API */
reply = redisCommand(c,"SET %b %b", "bar", (size_t) 3, "hello", (size_t) 5);
printf("SET (binary API): %s\n", reply->str);
freeReplyObject(reply);
/* Try a GET and two INCR */
reply = redisCommand(c,"GET foo");
printf("GET foo: %s\n", reply->str);
freeReplyObject(reply);
reply = redisCommand(c,"INCR counter");
printf("INCR counter: %lld\n", reply->integer);
freeReplyObject(reply);
/* again ... */
reply = redisCommand(c,"INCR counter");
printf("INCR counter: %lld\n", reply->integer);
freeReplyObject(reply);
/* Create a list of numbers, from 0 to 9 */
reply = redisCommand(c,"DEL mylist");
freeReplyObject(reply);
for (j = 0; j < 10; j++) {
char buf[64];
snprintf(buf,64,"%u",j);
reply = redisCommand(c,"LPUSH mylist element-%s", buf);
freeReplyObject(reply);
}
/* Let's check what we have inside the list */
reply = redisCommand(c,"LRANGE mylist 0 -1");
if (reply->type == REDIS_REPLY_ARRAY) {
for (j = 0; j < reply->elements; j++) {
printf("%u) %s\n", j, reply->element[j]->str);
}
}
freeReplyObject(reply);
/* Disconnects and frees the context */
redisFree(c);
return 0;
}
+15 -2
View File
@@ -5,14 +5,27 @@
#include <hiredis.h>
int main(int argc, char **argv) {
unsigned int j;
unsigned int j, isunix = 0;
redisContext *c;
redisReply *reply;
const char *hostname = (argc > 1) ? argv[1] : "127.0.0.1";
if (argc > 2) {
if (*argv[2] == 'u' || *argv[2] == 'U') {
isunix = 1;
/* in this case, host is the path to the unix socket */
printf("Will connect to unix socket @%s\n", hostname);
}
}
int port = (argc > 2) ? atoi(argv[2]) : 6379;
struct timeval timeout = { 1, 500000 }; // 1.5 seconds
c = redisConnectWithTimeout(hostname, port, timeout);
if (isunix) {
c = redisConnectUnixWithTimeout(hostname, timeout);
} else {
c = redisConnectWithTimeout(hostname, port, timeout);
}
if (c == NULL || c->err) {
if (c) {
printf("Connection error: %s\n", c->errstr);
+134 -122
View File
@@ -34,7 +34,6 @@
#include "fmacros.h"
#include <string.h>
#include <stdlib.h>
#include <unistd.h>
#include <assert.h>
#include <errno.h>
#include <ctype.h>
@@ -42,10 +41,20 @@
#include "hiredis.h"
#include "net.h"
#include "sds.h"
#include "async.h"
#include "win32.h"
static redisContextFuncs redisContextDefaultFuncs = {
.free_privdata = NULL,
.async_read = redisAsyncRead,
.async_write = redisAsyncWrite,
.read = redisNetRead,
.write = redisNetWrite
};
static redisReply *createReplyObject(int type);
static void *createStringObject(const redisReadTask *task, char *str, size_t len);
static void *createArrayObject(const redisReadTask *task, int elements);
static void *createArrayObject(const redisReadTask *task, size_t elements);
static void *createIntegerObject(const redisReadTask *task, long long value);
static void *createDoubleObject(const redisReadTask *task, double value, char *str, size_t len);
static void *createNilObject(const redisReadTask *task);
@@ -112,21 +121,34 @@ static void *createStringObject(const redisReadTask *task, char *str, size_t len
if (r == NULL)
return NULL;
buf = malloc(len+1);
if (buf == NULL) {
freeReplyObject(r);
return NULL;
}
assert(task->type == REDIS_REPLY_ERROR ||
task->type == REDIS_REPLY_STATUS ||
task->type == REDIS_REPLY_STRING);
task->type == REDIS_REPLY_STRING ||
task->type == REDIS_REPLY_VERB);
/* Copy string value */
memcpy(buf,str,len);
buf[len] = '\0';
if (task->type == REDIS_REPLY_VERB) {
buf = malloc(len-4+1); /* Skip 4 bytes of verbatim type header. */
if (buf == NULL) {
freeReplyObject(r);
return NULL;
}
memcpy(r->vtype,str,3);
r->vtype[3] = '\0';
memcpy(buf,str+4,len-4);
buf[len-4] = '\0';
r->len = len-4;
} else {
buf = malloc(len+1);
if (buf == NULL) {
freeReplyObject(r);
return NULL;
}
memcpy(buf,str,len);
buf[len] = '\0';
r->len = len;
}
r->str = buf;
r->len = len;
if (task->parent) {
parent = task->parent->obj;
@@ -138,7 +160,7 @@ static void *createStringObject(const redisReadTask *task, char *str, size_t len
return r;
}
static void *createArrayObject(const redisReadTask *task, int elements) {
static void *createArrayObject(const redisReadTask *task, size_t elements) {
redisReply *r, *parent;
r = createReplyObject(task->type);
@@ -649,29 +671,30 @@ redisReader *redisReaderCreate(void) {
return redisReaderCreateWithFunctions(&defaultFunctions);
}
static redisContext *redisContextInit(void) {
static redisContext *redisContextInit(const redisOptions *options) {
redisContext *c;
c = calloc(1,sizeof(redisContext));
c = calloc(1, sizeof(*c));
if (c == NULL)
return NULL;
c->funcs = &redisContextDefaultFuncs;
c->obuf = sdsempty();
c->reader = redisReaderCreate();
c->fd = REDIS_INVALID_FD;
if (c->obuf == NULL || c->reader == NULL) {
redisFree(c);
return NULL;
}
(void)options; /* options are used in other functions */
return c;
}
void redisFree(redisContext *c) {
if (c == NULL)
return;
if (c->fd > 0)
close(c->fd);
redisNetClose(c);
sdsfree(c->obuf);
redisReaderFree(c->reader);
@@ -680,12 +703,16 @@ void redisFree(redisContext *c) {
free(c->unix_sock.path);
free(c->timeout);
free(c->saddr);
if (c->funcs->free_privdata) {
c->funcs->free_privdata(c->privdata);
}
memset(c, 0xff, sizeof(*c));
free(c);
}
int redisFreeKeepFd(redisContext *c) {
int fd = c->fd;
c->fd = -1;
redisFD redisFreeKeepFd(redisContext *c) {
redisFD fd = c->fd;
c->fd = REDIS_INVALID_FD;
redisFree(c);
return fd;
}
@@ -694,10 +721,13 @@ int redisReconnect(redisContext *c) {
c->err = 0;
memset(c->errstr, '\0', strlen(c->errstr));
if (c->fd > 0) {
close(c->fd);
if (c->privdata && c->funcs->free_privdata) {
c->funcs->free_privdata(c->privdata);
c->privdata = NULL;
}
redisNetClose(c);
sdsfree(c->obuf);
redisReaderFree(c->reader);
@@ -718,112 +748,107 @@ int redisReconnect(redisContext *c) {
return REDIS_ERR;
}
redisContext *redisConnectWithOptions(const redisOptions *options) {
redisContext *c = redisContextInit(options);
if (c == NULL) {
return NULL;
}
if (!(options->options & REDIS_OPT_NONBLOCK)) {
c->flags |= REDIS_BLOCK;
}
if (options->options & REDIS_OPT_REUSEADDR) {
c->flags |= REDIS_REUSEADDR;
}
if (options->options & REDIS_OPT_NOAUTOFREE) {
c->flags |= REDIS_NO_AUTO_FREE;
}
if (options->type == REDIS_CONN_TCP) {
redisContextConnectBindTcp(c, options->endpoint.tcp.ip,
options->endpoint.tcp.port, options->timeout,
options->endpoint.tcp.source_addr);
} else if (options->type == REDIS_CONN_UNIX) {
redisContextConnectUnix(c, options->endpoint.unix_socket,
options->timeout);
} else if (options->type == REDIS_CONN_USERFD) {
c->fd = options->endpoint.fd;
c->flags |= REDIS_CONNECTED;
} else {
// Unknown type - FIXME - FREE
return NULL;
}
if (options->timeout != NULL && (c->flags & REDIS_BLOCK) && c->fd != REDIS_INVALID_FD) {
redisContextSetTimeout(c, *options->timeout);
}
return c;
}
/* Connect to a Redis instance. On error the field error in the returned
* context will be set to the return value of the error function.
* When no set of reply functions is given, the default set will be used. */
redisContext *redisConnect(const char *ip, int port) {
redisContext *c;
c = redisContextInit();
if (c == NULL)
return NULL;
c->flags |= REDIS_BLOCK;
redisContextConnectTcp(c,ip,port,NULL);
return c;
redisOptions options = {0};
REDIS_OPTIONS_SET_TCP(&options, ip, port);
return redisConnectWithOptions(&options);
}
redisContext *redisConnectWithTimeout(const char *ip, int port, const struct timeval tv) {
redisContext *c;
c = redisContextInit();
if (c == NULL)
return NULL;
c->flags |= REDIS_BLOCK;
redisContextConnectTcp(c,ip,port,&tv);
return c;
redisOptions options = {0};
REDIS_OPTIONS_SET_TCP(&options, ip, port);
options.timeout = &tv;
return redisConnectWithOptions(&options);
}
redisContext *redisConnectNonBlock(const char *ip, int port) {
redisContext *c;
c = redisContextInit();
if (c == NULL)
return NULL;
c->flags &= ~REDIS_BLOCK;
redisContextConnectTcp(c,ip,port,NULL);
return c;
redisOptions options = {0};
REDIS_OPTIONS_SET_TCP(&options, ip, port);
options.options |= REDIS_OPT_NONBLOCK;
return redisConnectWithOptions(&options);
}
redisContext *redisConnectBindNonBlock(const char *ip, int port,
const char *source_addr) {
redisContext *c = redisContextInit();
if (c == NULL)
return NULL;
c->flags &= ~REDIS_BLOCK;
redisContextConnectBindTcp(c,ip,port,NULL,source_addr);
return c;
redisOptions options = {0};
REDIS_OPTIONS_SET_TCP(&options, ip, port);
options.endpoint.tcp.source_addr = source_addr;
options.options |= REDIS_OPT_NONBLOCK;
return redisConnectWithOptions(&options);
}
redisContext *redisConnectBindNonBlockWithReuse(const char *ip, int port,
const char *source_addr) {
redisContext *c = redisContextInit();
if (c == NULL)
return NULL;
c->flags &= ~REDIS_BLOCK;
c->flags |= REDIS_REUSEADDR;
redisContextConnectBindTcp(c,ip,port,NULL,source_addr);
return c;
redisOptions options = {0};
REDIS_OPTIONS_SET_TCP(&options, ip, port);
options.endpoint.tcp.source_addr = source_addr;
options.options |= REDIS_OPT_NONBLOCK|REDIS_OPT_REUSEADDR;
return redisConnectWithOptions(&options);
}
redisContext *redisConnectUnix(const char *path) {
redisContext *c;
c = redisContextInit();
if (c == NULL)
return NULL;
c->flags |= REDIS_BLOCK;
redisContextConnectUnix(c,path,NULL);
return c;
redisOptions options = {0};
REDIS_OPTIONS_SET_UNIX(&options, path);
return redisConnectWithOptions(&options);
}
redisContext *redisConnectUnixWithTimeout(const char *path, const struct timeval tv) {
redisContext *c;
c = redisContextInit();
if (c == NULL)
return NULL;
c->flags |= REDIS_BLOCK;
redisContextConnectUnix(c,path,&tv);
return c;
redisOptions options = {0};
REDIS_OPTIONS_SET_UNIX(&options, path);
options.timeout = &tv;
return redisConnectWithOptions(&options);
}
redisContext *redisConnectUnixNonBlock(const char *path) {
redisContext *c;
c = redisContextInit();
if (c == NULL)
return NULL;
c->flags &= ~REDIS_BLOCK;
redisContextConnectUnix(c,path,NULL);
return c;
redisOptions options = {0};
REDIS_OPTIONS_SET_UNIX(&options, path);
options.options |= REDIS_OPT_NONBLOCK;
return redisConnectWithOptions(&options);
}
redisContext *redisConnectFd(int fd) {
redisContext *c;
c = redisContextInit();
if (c == NULL)
return NULL;
c->fd = fd;
c->flags |= REDIS_BLOCK | REDIS_CONNECTED;
return c;
redisContext *redisConnectFd(redisFD fd) {
redisOptions options = {0};
options.type = REDIS_CONN_USERFD;
options.endpoint.fd = fd;
return redisConnectWithOptions(&options);
}
/* Set read/write timeout on a blocking socket. */
@@ -853,22 +878,15 @@ int redisBufferRead(redisContext *c) {
if (c->err)
return REDIS_ERR;
nread = read(c->fd,buf,sizeof(buf));
if (nread == -1) {
if ((errno == EAGAIN && !(c->flags & REDIS_BLOCK)) || (errno == EINTR)) {
/* Try again later */
nread = c->funcs->read(c, buf, sizeof(buf));
if (nread > 0) {
if (redisReaderFeed(c->reader, buf, nread) != REDIS_OK) {
__redisSetError(c, c->reader->err, c->reader->errstr);
return REDIS_ERR;
} else {
__redisSetError(c,REDIS_ERR_IO,NULL);
return REDIS_ERR;
}
} else if (nread == 0) {
__redisSetError(c,REDIS_ERR_EOF,"Server closed the connection");
} else if (nread < 0) {
return REDIS_ERR;
} else {
if (redisReaderFeed(c->reader,buf,nread) != REDIS_OK) {
__redisSetError(c,c->reader->err,c->reader->errstr);
return REDIS_ERR;
}
}
return REDIS_OK;
}
@@ -883,21 +901,15 @@ int redisBufferRead(redisContext *c) {
* c->errstr to hold the appropriate error string.
*/
int redisBufferWrite(redisContext *c, int *done) {
int nwritten;
/* Return early when the context has seen an error. */
if (c->err)
return REDIS_ERR;
if (sdslen(c->obuf) > 0) {
nwritten = write(c->fd,c->obuf,sdslen(c->obuf));
if (nwritten == -1) {
if ((errno == EAGAIN && !(c->flags & REDIS_BLOCK)) || (errno == EINTR)) {
/* Try again later */
} else {
__redisSetError(c,REDIS_ERR_IO,NULL);
return REDIS_ERR;
}
int nwritten = c->funcs->write(c);
if (nwritten < 0) {
return REDIS_ERR;
} else if (nwritten > 0) {
if (nwritten == (signed)sdslen(c->obuf)) {
sdsfree(c->obuf);
+99 -4
View File
@@ -35,7 +35,11 @@
#define __HIREDIS_H
#include "read.h"
#include <stdarg.h> /* for va_list */
#ifndef _MSC_VER
#include <sys/time.h> /* for struct timeval */
#else
struct timeval; /* forward declaration */
#endif
#include <stdint.h> /* uintXX_t, etc */
#include "sds.h" /* for sds */
@@ -74,6 +78,12 @@
/* Flag that is set when we should set SO_REUSEADDR before calling bind() */
#define REDIS_REUSEADDR 0x80
/**
* Flag that indicates the user does not want the context to
* be automatically freed upon error
*/
#define REDIS_NO_AUTO_FREE 0x200
#define REDIS_KEEPALIVE_INTERVAL 15 /* seconds */
/* number of times we retry to connect in the case of EADDRNOTAVAIL and
@@ -92,6 +102,8 @@ typedef struct redisReply {
size_t len; /* Length of string */
char *str; /* Used for REDIS_REPLY_ERROR, REDIS_REPLY_STRING
and REDIS_REPLY_DOUBLE (in additionl to dval). */
char vtype[4]; /* Used for REDIS_REPLY_VERB, contains the null
terminated 3 character content type, such as "txt". */
size_t elements; /* number of elements, for REDIS_REPLY_ARRAY */
struct redisReply **element; /* elements vector for REDIS_REPLY_ARRAY */
} redisReply;
@@ -111,14 +123,93 @@ void redisFreeSdsCommand(sds cmd);
enum redisConnectionType {
REDIS_CONN_TCP,
REDIS_CONN_UNIX
REDIS_CONN_UNIX,
REDIS_CONN_USERFD
};
struct redisSsl;
#define REDIS_OPT_NONBLOCK 0x01
#define REDIS_OPT_REUSEADDR 0x02
/**
* Don't automatically free the async object on a connection failure,
* or other implicit conditions. Only free on an explicit call to disconnect() or free()
*/
#define REDIS_OPT_NOAUTOFREE 0x04
/* In Unix systems a file descriptor is a regular signed int, with -1
* representing an invalid descriptor. In Windows it is a SOCKET
* (32- or 64-bit unsigned integer depending on the architecture), where
* all bits set (~0) is INVALID_SOCKET. */
#ifndef _WIN32
typedef int redisFD;
#define REDIS_INVALID_FD -1
#else
#ifdef _WIN64
typedef unsigned long long redisFD; /* SOCKET = 64-bit UINT_PTR */
#else
typedef unsigned long redisFD; /* SOCKET = 32-bit UINT_PTR */
#endif
#define REDIS_INVALID_FD ((redisFD)(~0)) /* INVALID_SOCKET */
#endif
typedef struct {
/*
* the type of connection to use. This also indicates which
* `endpoint` member field to use
*/
int type;
/* bit field of REDIS_OPT_xxx */
int options;
/* timeout value. if NULL, no timeout is used */
const struct timeval *timeout;
union {
/** use this field for tcp/ip connections */
struct {
const char *source_addr;
const char *ip;
int port;
} tcp;
/** use this field for unix domain sockets */
const char *unix_socket;
/**
* use this field to have hiredis operate an already-open
* file descriptor */
redisFD fd;
} endpoint;
} redisOptions;
/**
* Helper macros to initialize options to their specified fields.
*/
#define REDIS_OPTIONS_SET_TCP(opts, ip_, port_) \
(opts)->type = REDIS_CONN_TCP; \
(opts)->endpoint.tcp.ip = ip_; \
(opts)->endpoint.tcp.port = port_;
#define REDIS_OPTIONS_SET_UNIX(opts, path) \
(opts)->type = REDIS_CONN_UNIX; \
(opts)->endpoint.unix_socket = path;
struct redisAsyncContext;
struct redisContext;
typedef struct redisContextFuncs {
void (*free_privdata)(void *);
void (*async_read)(struct redisAsyncContext *);
void (*async_write)(struct redisAsyncContext *);
int (*read)(struct redisContext *, char *, size_t);
int (*write)(struct redisContext *);
} redisContextFuncs;
/* Context for a connection to Redis */
typedef struct redisContext {
const redisContextFuncs *funcs; /* Function table */
int err; /* Error flags, 0 when there is no error */
char errstr[128]; /* String representation of error when applicable */
int fd;
redisFD fd;
int flags;
char *obuf; /* Write buffer */
redisReader *reader; /* Protocol reader */
@@ -139,8 +230,12 @@ typedef struct redisContext {
/* For non-blocking connect */
struct sockadr *saddr;
size_t addrlen;
/* Additional private data for hiredis addons such as SSL */
void *privdata;
} redisContext;
redisContext *redisConnectWithOptions(const redisOptions *options);
redisContext *redisConnect(const char *ip, int port);
redisContext *redisConnectWithTimeout(const char *ip, int port, const struct timeval tv);
redisContext *redisConnectNonBlock(const char *ip, int port);
@@ -151,7 +246,7 @@ redisContext *redisConnectBindNonBlockWithReuse(const char *ip, int port,
redisContext *redisConnectUnix(const char *path);
redisContext *redisConnectUnixWithTimeout(const char *path, const struct timeval tv);
redisContext *redisConnectUnixNonBlock(const char *path);
redisContext *redisConnectFd(int fd);
redisContext *redisConnectFd(redisFD fd);
/**
* Reconnect the given context using the saved information.
@@ -167,7 +262,7 @@ int redisReconnect(redisContext *c);
int redisSetTimeout(redisContext *c, const struct timeval tv);
int redisEnableKeepAlive(redisContext *c);
void redisFree(redisContext *c);
int redisFreeKeepFd(redisContext *c);
redisFD redisFreeKeepFd(redisContext *c);
int redisBufferRead(redisContext *c);
int redisBufferWrite(redisContext *c, int *done);
+11
View File
@@ -0,0 +1,11 @@
prefix=@CMAKE_INSTALL_PREFIX@
exec_prefix=${prefix}
libdir=${exec_prefix}/lib
includedir=${prefix}/include
pkgincludedir=${includedir}/hiredis
Name: hiredis
Description: Minimalistic C client library for Redis.
Version: @PROJECT_VERSION@
Libs: -L${libdir} -lhiredis
Cflags: -I${pkgincludedir} -D_FILE_OFFSET_BITS=64
+53
View File
@@ -0,0 +1,53 @@
/*
* Copyright (c) 2019, Redis Labs
*
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions are met:
*
* * Redistributions of source code must retain the above copyright notice,
* this list of conditions and the following disclaimer.
* * Redistributions in binary form must reproduce the above copyright
* notice, this list of conditions and the following disclaimer in the
* documentation and/or other materials provided with the distribution.
* * Neither the name of Redis nor the names of its contributors may be used
* to endorse or promote products derived from this software without
* specific prior written permission.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
* AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
* ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
* LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
* CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
* SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
* INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
* CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
* ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
* POSSIBILITY OF SUCH DAMAGE.
*/
#ifndef __HIREDIS_SSL_H
#define __HIREDIS_SSL_H
/* This is the underlying struct for SSL in ssl.h, which is not included to
* keep build dependencies short here.
*/
struct ssl_st;
/**
* Secure the connection using SSL. This should be done before any command is
* executed on the connection.
*/
int redisSecureConnection(redisContext *c, const char *capath, const char *certpath,
const char *keypath, const char *servername);
/**
* Initiate SSL/TLS negotiation on a provided context.
*/
int redisInitiateSSL(redisContext *c, struct ssl_st *ssl);
#endif /* __HIREDIS_SSL_H */
+12
View File
@@ -0,0 +1,12 @@
prefix=@CMAKE_INSTALL_PREFIX@
exec_prefix=${prefix}
libdir=${exec_prefix}/lib
includedir=${prefix}/include
pkgincludedir=${includedir}/hiredis
Name: hiredis_ssl
Description: SSL Support for hiredis.
Version: @PROJECT_VERSION@
Requires: hiredis
Libs: -L${libdir} -lhiredis_ssl
Libs.private: -lssl -lcrypto
+87 -35
View File
@@ -34,36 +34,64 @@
#include "fmacros.h"
#include <sys/types.h>
#include <sys/socket.h>
#include <sys/select.h>
#include <sys/un.h>
#include <netinet/in.h>
#include <netinet/tcp.h>
#include <arpa/inet.h>
#include <unistd.h>
#include <fcntl.h>
#include <string.h>
#include <netdb.h>
#include <errno.h>
#include <stdarg.h>
#include <stdio.h>
#include <poll.h>
#include <limits.h>
#include <stdlib.h>
#include "net.h"
#include "sds.h"
#include "sockcompat.h"
#include "win32.h"
/* Defined in hiredis.c */
void __redisSetError(redisContext *c, int type, const char *str);
static void redisContextCloseFd(redisContext *c) {
if (c && c->fd >= 0) {
void redisNetClose(redisContext *c) {
if (c && c->fd != REDIS_INVALID_FD) {
close(c->fd);
c->fd = -1;
c->fd = REDIS_INVALID_FD;
}
}
int redisNetRead(redisContext *c, char *buf, size_t bufcap) {
int nread = recv(c->fd, buf, bufcap, 0);
if (nread == -1) {
if ((errno == EWOULDBLOCK && !(c->flags & REDIS_BLOCK)) || (errno == EINTR)) {
/* Try again later */
return 0;
} else if(errno == ETIMEDOUT && (c->flags & REDIS_BLOCK)) {
/* especially in windows */
__redisSetError(c, REDIS_ERR_TIMEOUT, "recv timeout");
return -1;
} else {
__redisSetError(c, REDIS_ERR_IO, NULL);
return -1;
}
} else if (nread == 0) {
__redisSetError(c, REDIS_ERR_EOF, "Server closed the connection");
return -1;
} else {
return nread;
}
}
int redisNetWrite(redisContext *c) {
int nwritten = send(c->fd, c->obuf, sdslen(c->obuf), 0);
if (nwritten < 0) {
if ((errno == EWOULDBLOCK && !(c->flags & REDIS_BLOCK)) || (errno == EINTR)) {
/* Try again later */
} else {
__redisSetError(c, REDIS_ERR_IO, NULL);
return -1;
}
}
return nwritten;
}
static void __redisSetErrorFromErrno(redisContext *c, int type, const char *prefix) {
int errorno = errno; /* snprintf() may change errno */
char buf[128] = { 0 };
@@ -79,15 +107,15 @@ static int redisSetReuseAddr(redisContext *c) {
int on = 1;
if (setsockopt(c->fd, SOL_SOCKET, SO_REUSEADDR, &on, sizeof(on)) == -1) {
__redisSetErrorFromErrno(c,REDIS_ERR_IO,NULL);
redisContextCloseFd(c);
redisNetClose(c);
return REDIS_ERR;
}
return REDIS_OK;
}
static int redisCreateSocket(redisContext *c, int type) {
int s;
if ((s = socket(type, SOCK_STREAM, 0)) == -1) {
redisFD s;
if ((s = socket(type, SOCK_STREAM, 0)) == REDIS_INVALID_FD) {
__redisSetErrorFromErrno(c,REDIS_ERR_IO,NULL);
return REDIS_ERR;
}
@@ -101,6 +129,7 @@ static int redisCreateSocket(redisContext *c, int type) {
}
static int redisSetBlocking(redisContext *c, int blocking) {
#ifndef _WIN32
int flags;
/* Set the socket nonblocking.
@@ -108,7 +137,7 @@ static int redisSetBlocking(redisContext *c, int blocking) {
* interrupted by a signal. */
if ((flags = fcntl(c->fd, F_GETFL)) == -1) {
__redisSetErrorFromErrno(c,REDIS_ERR_IO,"fcntl(F_GETFL)");
redisContextCloseFd(c);
redisNetClose(c);
return REDIS_ERR;
}
@@ -119,15 +148,23 @@ static int redisSetBlocking(redisContext *c, int blocking) {
if (fcntl(c->fd, F_SETFL, flags) == -1) {
__redisSetErrorFromErrno(c,REDIS_ERR_IO,"fcntl(F_SETFL)");
redisContextCloseFd(c);
redisNetClose(c);
return REDIS_ERR;
}
#else
u_long mode = blocking ? 0 : 1;
if (ioctl(c->fd, FIONBIO, &mode) == -1) {
__redisSetErrorFromErrno(c, REDIS_ERR_IO, "ioctl(FIONBIO)");
redisNetClose(c);
return REDIS_ERR;
}
#endif /* _WIN32 */
return REDIS_OK;
}
int redisKeepAlive(redisContext *c, int interval) {
int val = 1;
int fd = c->fd;
redisFD fd = c->fd;
if (setsockopt(fd, SOL_SOCKET, SO_KEEPALIVE, &val, sizeof(val)) == -1){
__redisSetError(c,REDIS_ERR_OTHER,strerror(errno));
@@ -170,7 +207,7 @@ static int redisSetTcpNoDelay(redisContext *c) {
int yes = 1;
if (setsockopt(c->fd, IPPROTO_TCP, TCP_NODELAY, &yes, sizeof(yes)) == -1) {
__redisSetErrorFromErrno(c,REDIS_ERR_IO,"setsockopt(TCP_NODELAY)");
redisContextCloseFd(c);
redisNetClose(c);
return REDIS_ERR;
}
return REDIS_OK;
@@ -212,12 +249,12 @@ static int redisContextWaitReady(redisContext *c, long msec) {
if ((res = poll(wfd, 1, msec)) == -1) {
__redisSetErrorFromErrno(c, REDIS_ERR_IO, "poll(2)");
redisContextCloseFd(c);
redisNetClose(c);
return REDIS_ERR;
} else if (res == 0) {
errno = ETIMEDOUT;
__redisSetErrorFromErrno(c,REDIS_ERR_IO,NULL);
redisContextCloseFd(c);
redisNetClose(c);
return REDIS_ERR;
}
@@ -230,7 +267,7 @@ static int redisContextWaitReady(redisContext *c, long msec) {
}
__redisSetErrorFromErrno(c,REDIS_ERR_IO,NULL);
redisContextCloseFd(c);
redisNetClose(c);
return REDIS_ERR;
}
@@ -277,11 +314,18 @@ int redisCheckSocketError(redisContext *c) {
}
int redisContextSetTimeout(redisContext *c, const struct timeval tv) {
if (setsockopt(c->fd,SOL_SOCKET,SO_RCVTIMEO,&tv,sizeof(tv)) == -1) {
const void *to_ptr = &tv;
size_t to_sz = sizeof(tv);
#ifdef _WIN32
DWORD timeout_msec = tv.tv_sec * 1000 + tv.tv_usec / 1000;
to_ptr = &timeout_msec;
to_sz = sizeof(timeout_msec);
#endif
if (setsockopt(c->fd,SOL_SOCKET,SO_RCVTIMEO,to_ptr,to_sz) == -1) {
__redisSetErrorFromErrno(c,REDIS_ERR_IO,"setsockopt(SO_RCVTIMEO)");
return REDIS_ERR;
}
if (setsockopt(c->fd,SOL_SOCKET,SO_SNDTIMEO,&tv,sizeof(tv)) == -1) {
if (setsockopt(c->fd,SOL_SOCKET,SO_SNDTIMEO,to_ptr,to_sz) == -1) {
__redisSetErrorFromErrno(c,REDIS_ERR_IO,"setsockopt(SO_SNDTIMEO)");
return REDIS_ERR;
}
@@ -291,7 +335,8 @@ int redisContextSetTimeout(redisContext *c, const struct timeval tv) {
static int _redisContextConnectTcp(redisContext *c, const char *addr, int port,
const struct timeval *timeout,
const char *source_addr) {
int s, rv, n;
redisFD s;
int rv, n;
char _port[6]; /* strlen("65535"); */
struct addrinfo hints, *servinfo, *bservinfo, *p, *b;
int blocking = (c->flags & REDIS_BLOCK);
@@ -360,7 +405,7 @@ static int _redisContextConnectTcp(redisContext *c, const char *addr, int port,
}
for (p = servinfo; p != NULL; p = p->ai_next) {
addrretry:
if ((s = socket(p->ai_family,p->ai_socktype,p->ai_protocol)) == -1)
if ((s = socket(p->ai_family,p->ai_socktype,p->ai_protocol)) == REDIS_INVALID_FD)
continue;
c->fd = s;
@@ -401,16 +446,14 @@ addrretry:
}
/* For repeat connection */
if (c->saddr) {
free(c->saddr);
}
free(c->saddr);
c->saddr = malloc(p->ai_addrlen);
memcpy(c->saddr, p->ai_addr, p->ai_addrlen);
c->addrlen = p->ai_addrlen;
if (connect(s,p->ai_addr,p->ai_addrlen) == -1) {
if (errno == EHOSTUNREACH) {
redisContextCloseFd(c);
redisNetClose(c);
continue;
} else if (errno == EINPROGRESS) {
if (blocking) {
@@ -424,7 +467,7 @@ addrretry:
if (++reuses >= REDIS_CONNECT_RETRIES) {
goto error;
} else {
redisContextCloseFd(c);
redisNetClose(c);
goto addrretry;
}
} else {
@@ -471,8 +514,9 @@ int redisContextConnectBindTcp(redisContext *c, const char *addr, int port,
}
int redisContextConnectUnix(redisContext *c, const char *path, const struct timeval *timeout) {
#ifndef _WIN32
int blocking = (c->flags & REDIS_BLOCK);
struct sockaddr_un sa;
struct sockaddr_un *sa;
long timeout_msec = -1;
if (redisCreateSocket(c,AF_UNIX) < 0)
@@ -499,9 +543,11 @@ int redisContextConnectUnix(redisContext *c, const char *path, const struct time
if (redisContextTimeoutMsec(c,&timeout_msec) != REDIS_OK)
return REDIS_ERR;
sa.sun_family = AF_UNIX;
strncpy(sa.sun_path,path,sizeof(sa.sun_path)-1);
if (connect(c->fd, (struct sockaddr*)&sa, sizeof(sa)) == -1) {
sa = (struct sockaddr_un*)(c->saddr = malloc(sizeof(struct sockaddr_un)));
c->addrlen = sizeof(struct sockaddr_un);
sa->sun_family = AF_UNIX;
strncpy(sa->sun_path, path, sizeof(sa->sun_path) - 1);
if (connect(c->fd, (struct sockaddr*)sa, sizeof(*sa)) == -1) {
if (errno == EINPROGRESS && !blocking) {
/* This is ok. */
} else {
@@ -516,4 +562,10 @@ int redisContextConnectUnix(redisContext *c, const char *path, const struct time
c->flags |= REDIS_CONNECTED;
return REDIS_OK;
#else
/* We currently do not support Unix sockets for Windows. */
/* TODO(m): https://devblogs.microsoft.com/commandline/af_unix-comes-to-windows/ */
errno = EPROTONOSUPPORT;
return REDIS_ERR;
#endif /* _WIN32 */
}
+4
View File
@@ -37,6 +37,10 @@
#include "hiredis.h"
void redisNetClose(redisContext *c);
int redisNetRead(redisContext *c, char *buf, size_t bufcap);
int redisNetWrite(redisContext *c);
int redisCheckSocketError(redisContext *c);
int redisContextSetTimeout(redisContext *c, const struct timeval tv);
int redisContextConnectTcp(redisContext *c, const char *addr, int port, const struct timeval *timeout);
+22 -6
View File
@@ -31,10 +31,10 @@
#include "fmacros.h"
#include <string.h>
#include <strings.h>
#include <stdlib.h>
#ifndef _MSC_VER
#include <unistd.h>
#include <strings.h>
#endif
#include <assert.h>
#include <errno.h>
@@ -44,6 +44,7 @@
#include "read.h"
#include "sds.h"
#include "win32.h"
static void __redisReaderSetError(redisReader *r, int type, const char *str) {
size_t len;
@@ -294,9 +295,9 @@ static int processLineItem(redisReader *r) {
buf[len] = '\0';
if (strcasecmp(buf,",inf") == 0) {
d = 1.0/0.0; /* Positive infinite. */
d = INFINITY; /* Positive infinite. */
} else if (strcasecmp(buf,",-inf") == 0) {
d = -1.0/0.0; /* Nevative infinite. */
d = -INFINITY; /* Nevative infinite. */
} else {
d = strtod((char*)buf,&eptr);
if (buf[0] == '\0' || eptr[0] != '\0' || isnan(d)) {
@@ -379,10 +380,18 @@ static int processBulkItem(redisReader *r) {
/* Only continue when the buffer contains the entire bulk item. */
bytelen += len+2; /* include \r\n */
if (r->pos+bytelen <= r->len) {
if ((cur->type == REDIS_REPLY_VERB && len < 4) ||
(cur->type == REDIS_REPLY_VERB && s[5] != ':'))
{
__redisReaderSetError(r,REDIS_ERR_PROTOCOL,
"Verbatim string 4 bytes of content type are "
"missing or incorrectly encoded.");
return REDIS_ERR;
}
if (r->fn && r->fn->createString)
obj = r->fn->createString(cur,s+2,len);
else
obj = (void*)REDIS_REPLY_STRING;
obj = (void*)(long)cur->type;
success = 1;
}
}
@@ -430,7 +439,7 @@ static int processAggregateItem(redisReader *r) {
root = (r->ridx == 0);
if (elements < -1 || elements > INT_MAX) {
if (elements < -1 || (LLONG_MAX > SIZE_MAX && elements > SIZE_MAX)) {
__redisReaderSetError(r,REDIS_ERR_PROTOCOL,
"Multi-bulk length out of range");
return REDIS_ERR;
@@ -523,6 +532,9 @@ static int processItem(redisReader *r) {
case '#':
cur->type = REDIS_REPLY_BOOL;
break;
case '=':
cur->type = REDIS_REPLY_VERB;
break;
default:
__redisReaderSetErrorProtocolByte(r,*p);
return REDIS_ERR;
@@ -543,6 +555,7 @@ static int processItem(redisReader *r) {
case REDIS_REPLY_BOOL:
return processLineItem(r);
case REDIS_REPLY_STRING:
case REDIS_REPLY_VERB:
return processBulkItem(r);
case REDIS_REPLY_ARRAY:
case REDIS_REPLY_MAP:
@@ -657,8 +670,11 @@ int redisReaderGetReply(redisReader *r, void **reply) {
/* Emit a reply when there is one. */
if (r->ridx == -1) {
if (reply != NULL)
if (reply != NULL) {
*reply = r->reply;
} else if (r->reply != NULL && r->fn && r->fn->freeObject) {
r->fn->freeObject(r->reply);
}
r->reply = NULL;
}
return REDIS_OK;
+3 -2
View File
@@ -45,6 +45,7 @@
#define REDIS_ERR_EOF 3 /* End of file */
#define REDIS_ERR_PROTOCOL 4 /* Protocol error */
#define REDIS_ERR_OOM 5 /* Out of memory */
#define REDIS_ERR_TIMEOUT 6 /* Timed out */
#define REDIS_ERR_OTHER 2 /* Everything else... */
#define REDIS_REPLY_STRING 1
@@ -55,12 +56,12 @@
#define REDIS_REPLY_ERROR 6
#define REDIS_REPLY_DOUBLE 7
#define REDIS_REPLY_BOOL 8
#define REDIS_REPLY_VERB 9
#define REDIS_REPLY_MAP 9
#define REDIS_REPLY_SET 10
#define REDIS_REPLY_ATTR 11
#define REDIS_REPLY_PUSH 12
#define REDIS_REPLY_BIGNUM 13
#define REDIS_REPLY_VERB 14
#define REDIS_READER_MAX_BUF (1024*16) /* Default max unused reader buffer. */
@@ -79,7 +80,7 @@ typedef struct redisReadTask {
typedef struct redisReplyObjectFunctions {
void *(*createString)(const redisReadTask*, char*, size_t);
void *(*createArray)(const redisReadTask*, int);
void *(*createArray)(const redisReadTask*, size_t);
void *(*createInteger)(const redisReadTask*, long long);
void *(*createDouble)(const redisReadTask*, double, char*, size_t);
void *(*createNil)(const redisReadTask*);
+1 -1
View File
@@ -1035,7 +1035,7 @@ sds *sdssplitargs(const char *line, int *argc) {
s_free(vector);
return NULL;
}
vector = new_vector;
vector[*argc] = current;
(*argc)++;
+17 -14
View File
@@ -34,6 +34,9 @@
#define __SDS_H
#define SDS_MAX_PREALLOC (1024*1024)
#ifdef _MSC_VER
#define __attribute__(x)
#endif
#include <sys/types.h>
#include <stdarg.h>
@@ -143,20 +146,20 @@ static inline void sdssetlen(sds s, size_t newlen) {
case SDS_TYPE_5:
{
unsigned char *fp = ((unsigned char*)s)-1;
*fp = SDS_TYPE_5 | (newlen << SDS_TYPE_BITS);
*fp = (unsigned char)(SDS_TYPE_5 | (newlen << SDS_TYPE_BITS));
}
break;
case SDS_TYPE_8:
SDS_HDR(8,s)->len = newlen;
SDS_HDR(8,s)->len = (uint8_t)newlen;
break;
case SDS_TYPE_16:
SDS_HDR(16,s)->len = newlen;
SDS_HDR(16,s)->len = (uint16_t)newlen;
break;
case SDS_TYPE_32:
SDS_HDR(32,s)->len = newlen;
SDS_HDR(32,s)->len = (uint32_t)newlen;
break;
case SDS_TYPE_64:
SDS_HDR(64,s)->len = newlen;
SDS_HDR(64,s)->len = (uint64_t)newlen;
break;
}
}
@@ -167,21 +170,21 @@ static inline void sdsinclen(sds s, size_t inc) {
case SDS_TYPE_5:
{
unsigned char *fp = ((unsigned char*)s)-1;
unsigned char newlen = SDS_TYPE_5_LEN(flags)+inc;
unsigned char newlen = SDS_TYPE_5_LEN(flags)+(unsigned char)inc;
*fp = SDS_TYPE_5 | (newlen << SDS_TYPE_BITS);
}
break;
case SDS_TYPE_8:
SDS_HDR(8,s)->len += inc;
SDS_HDR(8,s)->len += (uint8_t)inc;
break;
case SDS_TYPE_16:
SDS_HDR(16,s)->len += inc;
SDS_HDR(16,s)->len += (uint16_t)inc;
break;
case SDS_TYPE_32:
SDS_HDR(32,s)->len += inc;
SDS_HDR(32,s)->len += (uint32_t)inc;
break;
case SDS_TYPE_64:
SDS_HDR(64,s)->len += inc;
SDS_HDR(64,s)->len += (uint64_t)inc;
break;
}
}
@@ -211,16 +214,16 @@ static inline void sdssetalloc(sds s, size_t newlen) {
/* Nothing to do, this type has no total allocation info. */
break;
case SDS_TYPE_8:
SDS_HDR(8,s)->alloc = newlen;
SDS_HDR(8,s)->alloc = (uint8_t)newlen;
break;
case SDS_TYPE_16:
SDS_HDR(16,s)->alloc = newlen;
SDS_HDR(16,s)->alloc = (uint16_t)newlen;
break;
case SDS_TYPE_32:
SDS_HDR(32,s)->alloc = newlen;
SDS_HDR(32,s)->alloc = (uint32_t)newlen;
break;
case SDS_TYPE_64:
SDS_HDR(64,s)->alloc = newlen;
SDS_HDR(64,s)->alloc = (uint64_t)newlen;
break;
}
}
+248
View File
@@ -0,0 +1,248 @@
/*
* Copyright (c) 2019, Marcus Geelnard <m at bitsnbites dot eu>
*
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions are met:
*
* * Redistributions of source code must retain the above copyright notice,
* this list of conditions and the following disclaimer.
* * Redistributions in binary form must reproduce the above copyright
* notice, this list of conditions and the following disclaimer in the
* documentation and/or other materials provided with the distribution.
* * Neither the name of Redis nor the names of its contributors may be used
* to endorse or promote products derived from this software without
* specific prior written permission.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
* AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
* ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
* LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
* CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
* SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
* INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
* CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
* ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
* POSSIBILITY OF SUCH DAMAGE.
*/
#define REDIS_SOCKCOMPAT_IMPLEMENTATION
#include "sockcompat.h"
#ifdef _WIN32
static int _wsaErrorToErrno(int err) {
switch (err) {
case WSAEWOULDBLOCK:
return EWOULDBLOCK;
case WSAEINPROGRESS:
return EINPROGRESS;
case WSAEALREADY:
return EALREADY;
case WSAENOTSOCK:
return ENOTSOCK;
case WSAEDESTADDRREQ:
return EDESTADDRREQ;
case WSAEMSGSIZE:
return EMSGSIZE;
case WSAEPROTOTYPE:
return EPROTOTYPE;
case WSAENOPROTOOPT:
return ENOPROTOOPT;
case WSAEPROTONOSUPPORT:
return EPROTONOSUPPORT;
case WSAEOPNOTSUPP:
return EOPNOTSUPP;
case WSAEAFNOSUPPORT:
return EAFNOSUPPORT;
case WSAEADDRINUSE:
return EADDRINUSE;
case WSAEADDRNOTAVAIL:
return EADDRNOTAVAIL;
case WSAENETDOWN:
return ENETDOWN;
case WSAENETUNREACH:
return ENETUNREACH;
case WSAENETRESET:
return ENETRESET;
case WSAECONNABORTED:
return ECONNABORTED;
case WSAECONNRESET:
return ECONNRESET;
case WSAENOBUFS:
return ENOBUFS;
case WSAEISCONN:
return EISCONN;
case WSAENOTCONN:
return ENOTCONN;
case WSAETIMEDOUT:
return ETIMEDOUT;
case WSAECONNREFUSED:
return ECONNREFUSED;
case WSAELOOP:
return ELOOP;
case WSAENAMETOOLONG:
return ENAMETOOLONG;
case WSAEHOSTUNREACH:
return EHOSTUNREACH;
case WSAENOTEMPTY:
return ENOTEMPTY;
default:
/* We just return a generic I/O error if we could not find a relevant error. */
return EIO;
}
}
static void _updateErrno(int success) {
errno = success ? 0 : _wsaErrorToErrno(WSAGetLastError());
}
static int _initWinsock() {
static int s_initialized = 0;
if (!s_initialized) {
static WSADATA wsadata;
int err = WSAStartup(MAKEWORD(2,2), &wsadata);
if (err != 0) {
errno = _wsaErrorToErrno(err);
return 0;
}
s_initialized = 1;
}
return 1;
}
int win32_getaddrinfo(const char *node, const char *service, const struct addrinfo *hints, struct addrinfo **res) {
/* Note: This function is likely to be called before other functions, so run init here. */
if (!_initWinsock()) {
return EAI_FAIL;
}
switch (getaddrinfo(node, service, hints, res)) {
case 0: return 0;
case WSATRY_AGAIN: return EAI_AGAIN;
case WSAEINVAL: return EAI_BADFLAGS;
case WSAEAFNOSUPPORT: return EAI_FAMILY;
case WSA_NOT_ENOUGH_MEMORY: return EAI_MEMORY;
case WSAHOST_NOT_FOUND: return EAI_NONAME;
case WSATYPE_NOT_FOUND: return EAI_SERVICE;
case WSAESOCKTNOSUPPORT: return EAI_SOCKTYPE;
default: return EAI_FAIL; /* Including WSANO_RECOVERY */
}
}
const char *win32_gai_strerror(int errcode) {
switch (errcode) {
case 0: errcode = 0; break;
case EAI_AGAIN: errcode = WSATRY_AGAIN; break;
case EAI_BADFLAGS: errcode = WSAEINVAL; break;
case EAI_FAMILY: errcode = WSAEAFNOSUPPORT; break;
case EAI_MEMORY: errcode = WSA_NOT_ENOUGH_MEMORY; break;
case EAI_NONAME: errcode = WSAHOST_NOT_FOUND; break;
case EAI_SERVICE: errcode = WSATYPE_NOT_FOUND; break;
case EAI_SOCKTYPE: errcode = WSAESOCKTNOSUPPORT; break;
default: errcode = WSANO_RECOVERY; break; /* Including EAI_FAIL */
}
return gai_strerror(errcode);
}
void win32_freeaddrinfo(struct addrinfo *res) {
freeaddrinfo(res);
}
SOCKET win32_socket(int domain, int type, int protocol) {
SOCKET s;
/* Note: This function is likely to be called before other functions, so run init here. */
if (!_initWinsock()) {
return INVALID_SOCKET;
}
_updateErrno((s = socket(domain, type, protocol)) != INVALID_SOCKET);
return s;
}
int win32_ioctl(SOCKET fd, unsigned long request, unsigned long *argp) {
int ret = ioctlsocket(fd, (long)request, argp);
_updateErrno(ret != SOCKET_ERROR);
return ret != SOCKET_ERROR ? ret : -1;
}
int win32_bind(SOCKET sockfd, const struct sockaddr *addr, socklen_t addrlen) {
int ret = bind(sockfd, addr, addrlen);
_updateErrno(ret != SOCKET_ERROR);
return ret != SOCKET_ERROR ? ret : -1;
}
int win32_connect(SOCKET sockfd, const struct sockaddr *addr, socklen_t addrlen) {
int ret = connect(sockfd, addr, addrlen);
_updateErrno(ret != SOCKET_ERROR);
/* For Winsock connect(), the WSAEWOULDBLOCK error means the same thing as
* EINPROGRESS for POSIX connect(), so we do that translation to keep POSIX
* logic consistent. */
if (errno == EWOULDBLOCK) {
errno = EINPROGRESS;
}
return ret != SOCKET_ERROR ? ret : -1;
}
int win32_getsockopt(SOCKET sockfd, int level, int optname, void *optval, socklen_t *optlen) {
int ret = 0;
if ((level == SOL_SOCKET) && ((optname == SO_RCVTIMEO) || (optname == SO_SNDTIMEO))) {
if (*optlen >= sizeof (struct timeval)) {
struct timeval *tv = optval;
DWORD timeout = 0;
socklen_t dwlen = 0;
ret = getsockopt(sockfd, level, optname, (char *)&timeout, &dwlen);
tv->tv_sec = timeout / 1000;
tv->tv_usec = (timeout * 1000) % 1000000;
} else {
ret = WSAEFAULT;
}
*optlen = sizeof (struct timeval);
} else {
ret = getsockopt(sockfd, level, optname, (char*)optval, optlen);
}
_updateErrno(ret != SOCKET_ERROR);
return ret != SOCKET_ERROR ? ret : -1;
}
int win32_setsockopt(SOCKET sockfd, int level, int optname, const void *optval, socklen_t optlen) {
int ret = 0;
if ((level == SOL_SOCKET) && ((optname == SO_RCVTIMEO) || (optname == SO_SNDTIMEO))) {
struct timeval *tv = optval;
DWORD timeout = tv->tv_sec * 1000 + tv->tv_usec / 1000;
ret = setsockopt(sockfd, level, optname, (const char*)&timeout, sizeof(DWORD));
} else {
ret = setsockopt(sockfd, level, optname, (const char*)optval, optlen);
}
_updateErrno(ret != SOCKET_ERROR);
return ret != SOCKET_ERROR ? ret : -1;
}
int win32_close(SOCKET fd) {
int ret = closesocket(fd);
_updateErrno(ret != SOCKET_ERROR);
return ret != SOCKET_ERROR ? ret : -1;
}
ssize_t win32_recv(SOCKET sockfd, void *buf, size_t len, int flags) {
int ret = recv(sockfd, (char*)buf, (int)len, flags);
_updateErrno(ret != SOCKET_ERROR);
return ret != SOCKET_ERROR ? ret : -1;
}
ssize_t win32_send(SOCKET sockfd, const void *buf, size_t len, int flags) {
int ret = send(sockfd, (const char*)buf, (int)len, flags);
_updateErrno(ret != SOCKET_ERROR);
return ret != SOCKET_ERROR ? ret : -1;
}
int win32_poll(struct pollfd *fds, nfds_t nfds, int timeout) {
int ret = WSAPoll(fds, nfds, timeout);
_updateErrno(ret != SOCKET_ERROR);
return ret != SOCKET_ERROR ? ret : -1;
}
#endif /* _WIN32 */
+91
View File
@@ -0,0 +1,91 @@
/*
* Copyright (c) 2019, Marcus Geelnard <m at bitsnbites dot eu>
*
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions are met:
*
* * Redistributions of source code must retain the above copyright notice,
* this list of conditions and the following disclaimer.
* * Redistributions in binary form must reproduce the above copyright
* notice, this list of conditions and the following disclaimer in the
* documentation and/or other materials provided with the distribution.
* * Neither the name of Redis nor the names of its contributors may be used
* to endorse or promote products derived from this software without
* specific prior written permission.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
* AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
* ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
* LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
* CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
* SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
* INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
* CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
* ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
* POSSIBILITY OF SUCH DAMAGE.
*/
#ifndef __SOCKCOMPAT_H
#define __SOCKCOMPAT_H
#ifndef _WIN32
/* For POSIX systems we use the standard BSD socket API. */
#include <unistd.h>
#include <sys/socket.h>
#include <sys/select.h>
#include <sys/un.h>
#include <netinet/in.h>
#include <netinet/tcp.h>
#include <arpa/inet.h>
#include <netdb.h>
#include <poll.h>
#else
/* For Windows we use winsock. */
#undef _WIN32_WINNT
#define _WIN32_WINNT 0x0600 /* To get WSAPoll etc. */
#include <winsock2.h>
#include <ws2tcpip.h>
#include <stddef.h>
#ifdef _MSC_VER
typedef signed long ssize_t;
#endif
/* Emulate the parts of the BSD socket API that we need (override the winsock signatures). */
int win32_getaddrinfo(const char *node, const char *service, const struct addrinfo *hints, struct addrinfo **res);
const char *win32_gai_strerror(int errcode);
void win32_freeaddrinfo(struct addrinfo *res);
SOCKET win32_socket(int domain, int type, int protocol);
int win32_ioctl(SOCKET fd, unsigned long request, unsigned long *argp);
int win32_bind(SOCKET sockfd, const struct sockaddr *addr, socklen_t addrlen);
int win32_connect(SOCKET sockfd, const struct sockaddr *addr, socklen_t addrlen);
int win32_getsockopt(SOCKET sockfd, int level, int optname, void *optval, socklen_t *optlen);
int win32_setsockopt(SOCKET sockfd, int level, int optname, const void *optval, socklen_t optlen);
int win32_close(SOCKET fd);
ssize_t win32_recv(SOCKET sockfd, void *buf, size_t len, int flags);
ssize_t win32_send(SOCKET sockfd, const void *buf, size_t len, int flags);
typedef ULONG nfds_t;
int win32_poll(struct pollfd *fds, nfds_t nfds, int timeout);
#ifndef REDIS_SOCKCOMPAT_IMPLEMENTATION
#define getaddrinfo(node, service, hints, res) win32_getaddrinfo(node, service, hints, res)
#undef gai_strerror
#define gai_strerror(errcode) win32_gai_strerror(errcode)
#define freeaddrinfo(res) win32_freeaddrinfo(res)
#define socket(domain, type, protocol) win32_socket(domain, type, protocol)
#define ioctl(fd, request, argp) win32_ioctl(fd, request, argp)
#define bind(sockfd, addr, addrlen) win32_bind(sockfd, addr, addrlen)
#define connect(sockfd, addr, addrlen) win32_connect(sockfd, addr, addrlen)
#define getsockopt(sockfd, level, optname, optval, optlen) win32_getsockopt(sockfd, level, optname, optval, optlen)
#define setsockopt(sockfd, level, optname, optval, optlen) win32_setsockopt(sockfd, level, optname, optval, optlen)
#define close(fd) win32_close(fd)
#define recv(sockfd, buf, len, flags) win32_recv(sockfd, buf, len, flags)
#define send(sockfd, buf, len, flags) win32_send(sockfd, buf, len, flags)
#define poll(fds, nfds, timeout) win32_poll(fds, nfds, timeout)
#endif /* REDIS_SOCKCOMPAT_IMPLEMENTATION */
#endif /* _WIN32 */
#endif /* __SOCKCOMPAT_H */
+448
View File
@@ -0,0 +1,448 @@
/*
* Copyright (c) 2009-2011, Salvatore Sanfilippo <antirez at gmail dot com>
* Copyright (c) 2010-2011, Pieter Noordhuis <pcnoordhuis at gmail dot com>
* Copyright (c) 2019, Redis Labs
*
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions are met:
*
* * Redistributions of source code must retain the above copyright notice,
* this list of conditions and the following disclaimer.
* * Redistributions in binary form must reproduce the above copyright
* notice, this list of conditions and the following disclaimer in the
* documentation and/or other materials provided with the distribution.
* * Neither the name of Redis nor the names of its contributors may be used
* to endorse or promote products derived from this software without
* specific prior written permission.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
* AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
* ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
* LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
* CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
* SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
* INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
* CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
* ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
* POSSIBILITY OF SUCH DAMAGE.
*/
#include "hiredis.h"
#include "async.h"
#include <assert.h>
#include <pthread.h>
#include <errno.h>
#include <string.h>
#include <openssl/ssl.h>
#include <openssl/err.h>
#include "async_private.h"
void __redisSetError(redisContext *c, int type, const char *str);
/* The SSL context is attached to SSL/TLS connections as a privdata. */
typedef struct redisSSLContext {
/**
* OpenSSL SSL_CTX; It is optional and will not be set when using
* user-supplied SSL.
*/
SSL_CTX *ssl_ctx;
/**
* OpenSSL SSL object.
*/
SSL *ssl;
/**
* SSL_write() requires to be called again with the same arguments it was
* previously called with in the event of an SSL_read/SSL_write situation
*/
size_t lastLen;
/** Whether the SSL layer requires read (possibly before a write) */
int wantRead;
/**
* Whether a write was requested prior to a read. If set, the write()
* should resume whenever a read takes place, if possible
*/
int pendingWrite;
} redisSSLContext;
/* Forward declaration */
redisContextFuncs redisContextSSLFuncs;
#ifdef HIREDIS_SSL_TRACE
/**
* Callback used for debugging
*/
static void sslLogCallback(const SSL *ssl, int where, int ret) {
const char *retstr = "";
int should_log = 1;
/* Ignore low-level SSL stuff */
if (where & SSL_CB_ALERT) {
should_log = 1;
}
if (where == SSL_CB_HANDSHAKE_START || where == SSL_CB_HANDSHAKE_DONE) {
should_log = 1;
}
if ((where & SSL_CB_EXIT) && ret == 0) {
should_log = 1;
}
if (!should_log) {
return;
}
retstr = SSL_alert_type_string(ret);
printf("ST(0x%x). %s. R(0x%x)%s\n", where, SSL_state_string_long(ssl), ret, retstr);
if (where == SSL_CB_HANDSHAKE_DONE) {
printf("Using SSL version %s. Cipher=%s\n", SSL_get_version(ssl), SSL_get_cipher_name(ssl));
}
}
#endif
/**
* OpenSSL global initialization and locking handling callbacks.
* Note that this is only required for OpenSSL < 1.1.0.
*/
#if OPENSSL_VERSION_NUMBER < 0x10100000L
#define HIREDIS_USE_CRYPTO_LOCKS
#endif
#ifdef HIREDIS_USE_CRYPTO_LOCKS
typedef pthread_mutex_t sslLockType;
static void sslLockInit(sslLockType *l) {
pthread_mutex_init(l, NULL);
}
static void sslLockAcquire(sslLockType *l) {
pthread_mutex_lock(l);
}
static void sslLockRelease(sslLockType *l) {
pthread_mutex_unlock(l);
}
static pthread_mutex_t *ossl_locks;
static void opensslDoLock(int mode, int lkid, const char *f, int line) {
sslLockType *l = ossl_locks + lkid;
if (mode & CRYPTO_LOCK) {
sslLockAcquire(l);
} else {
sslLockRelease(l);
}
(void)f;
(void)line;
}
static void initOpensslLocks(void) {
unsigned ii, nlocks;
if (CRYPTO_get_locking_callback() != NULL) {
/* Someone already set the callback before us. Don't destroy it! */
return;
}
nlocks = CRYPTO_num_locks();
ossl_locks = malloc(sizeof(*ossl_locks) * nlocks);
for (ii = 0; ii < nlocks; ii++) {
sslLockInit(ossl_locks + ii);
}
CRYPTO_set_locking_callback(opensslDoLock);
}
#endif /* HIREDIS_USE_CRYPTO_LOCKS */
/**
* SSL Connection initialization.
*/
static int redisSSLConnect(redisContext *c, SSL_CTX *ssl_ctx, SSL *ssl) {
if (c->privdata) {
__redisSetError(c, REDIS_ERR_OTHER, "redisContext was already associated");
return REDIS_ERR;
}
c->privdata = calloc(1, sizeof(redisSSLContext));
c->funcs = &redisContextSSLFuncs;
redisSSLContext *rssl = c->privdata;
rssl->ssl_ctx = ssl_ctx;
rssl->ssl = ssl;
SSL_set_mode(rssl->ssl, SSL_MODE_ACCEPT_MOVING_WRITE_BUFFER);
SSL_set_fd(rssl->ssl, c->fd);
SSL_set_connect_state(rssl->ssl);
ERR_clear_error();
int rv = SSL_connect(rssl->ssl);
if (rv == 1) {
return REDIS_OK;
}
rv = SSL_get_error(rssl->ssl, rv);
if (((c->flags & REDIS_BLOCK) == 0) &&
(rv == SSL_ERROR_WANT_READ || rv == SSL_ERROR_WANT_WRITE)) {
return REDIS_OK;
}
if (c->err == 0) {
char err[512];
if (rv == SSL_ERROR_SYSCALL)
snprintf(err,sizeof(err)-1,"SSL_connect failed: %s",strerror(errno));
else {
unsigned long e = ERR_peek_last_error();
snprintf(err,sizeof(err)-1,"SSL_connect failed: %s",
ERR_reason_error_string(e));
}
__redisSetError(c, REDIS_ERR_IO, err);
}
return REDIS_ERR;
}
int redisInitiateSSL(redisContext *c, SSL *ssl) {
return redisSSLConnect(c, NULL, ssl);
}
int redisSecureConnection(redisContext *c, const char *capath,
const char *certpath, const char *keypath, const char *servername) {
SSL_CTX *ssl_ctx = NULL;
SSL *ssl = NULL;
/* Initialize global OpenSSL stuff */
static int isInit = 0;
if (!isInit) {
isInit = 1;
SSL_library_init();
#ifdef HIREDIS_USE_CRYPTO_LOCKS
initOpensslLocks();
#endif
}
ssl_ctx = SSL_CTX_new(SSLv23_client_method());
if (!ssl_ctx) {
__redisSetError(c, REDIS_ERR_OTHER, "Failed to create SSL_CTX");
goto error;
}
#ifdef HIREDIS_SSL_TRACE
SSL_CTX_set_info_callback(ssl_ctx, sslLogCallback);
#endif
SSL_CTX_set_options(ssl_ctx, SSL_OP_NO_SSLv2 | SSL_OP_NO_SSLv3);
SSL_CTX_set_verify(ssl_ctx, SSL_VERIFY_PEER, NULL);
if ((certpath != NULL && keypath == NULL) || (keypath != NULL && certpath == NULL)) {
__redisSetError(c, REDIS_ERR_OTHER, "certpath and keypath must be specified together");
goto error;
}
if (capath) {
if (!SSL_CTX_load_verify_locations(ssl_ctx, capath, NULL)) {
__redisSetError(c, REDIS_ERR_OTHER, "Invalid CA certificate");
goto error;
}
}
if (certpath) {
if (!SSL_CTX_use_certificate_chain_file(ssl_ctx, certpath)) {
__redisSetError(c, REDIS_ERR_OTHER, "Invalid client certificate");
goto error;
}
if (!SSL_CTX_use_PrivateKey_file(ssl_ctx, keypath, SSL_FILETYPE_PEM)) {
__redisSetError(c, REDIS_ERR_OTHER, "Invalid client key");
goto error;
}
}
ssl = SSL_new(ssl_ctx);
if (!ssl) {
__redisSetError(c, REDIS_ERR_OTHER, "Couldn't create new SSL instance");
goto error;
}
if (servername) {
if (!SSL_set_tlsext_host_name(ssl, servername)) {
__redisSetError(c, REDIS_ERR_OTHER, "Couldn't set server name indication");
goto error;
}
}
return redisSSLConnect(c, ssl_ctx, ssl);
error:
if (ssl) SSL_free(ssl);
if (ssl_ctx) SSL_CTX_free(ssl_ctx);
return REDIS_ERR;
}
static int maybeCheckWant(redisSSLContext *rssl, int rv) {
/**
* If the error is WANT_READ or WANT_WRITE, the appropriate flags are set
* and true is returned. False is returned otherwise
*/
if (rv == SSL_ERROR_WANT_READ) {
rssl->wantRead = 1;
return 1;
} else if (rv == SSL_ERROR_WANT_WRITE) {
rssl->pendingWrite = 1;
return 1;
} else {
return 0;
}
}
/**
* Implementation of redisContextFuncs for SSL connections.
*/
static void redisSSLFreeContext(void *privdata){
redisSSLContext *rsc = privdata;
if (!rsc) return;
if (rsc->ssl) {
SSL_free(rsc->ssl);
rsc->ssl = NULL;
}
if (rsc->ssl_ctx) {
SSL_CTX_free(rsc->ssl_ctx);
rsc->ssl_ctx = NULL;
}
free(rsc);
}
static int redisSSLRead(redisContext *c, char *buf, size_t bufcap) {
redisSSLContext *rssl = c->privdata;
int nread = SSL_read(rssl->ssl, buf, bufcap);
if (nread > 0) {
return nread;
} else if (nread == 0) {
__redisSetError(c, REDIS_ERR_EOF, "Server closed the connection");
return -1;
} else {
int err = SSL_get_error(rssl->ssl, nread);
if (c->flags & REDIS_BLOCK) {
/**
* In blocking mode, we should never end up in a situation where
* we get an error without it being an actual error, except
* in the case of EINTR, which can be spuriously received from
* debuggers or whatever.
*/
if (errno == EINTR) {
return 0;
} else {
const char *msg = NULL;
if (errno == EAGAIN) {
msg = "Resource temporarily unavailable";
}
__redisSetError(c, REDIS_ERR_IO, msg);
return -1;
}
}
/**
* We can very well get an EWOULDBLOCK/EAGAIN, however
*/
if (maybeCheckWant(rssl, err)) {
return 0;
} else {
__redisSetError(c, REDIS_ERR_IO, NULL);
return -1;
}
}
}
static int redisSSLWrite(redisContext *c) {
redisSSLContext *rssl = c->privdata;
size_t len = rssl->lastLen ? rssl->lastLen : sdslen(c->obuf);
int rv = SSL_write(rssl->ssl, c->obuf, len);
if (rv > 0) {
rssl->lastLen = 0;
} else if (rv < 0) {
rssl->lastLen = len;
int err = SSL_get_error(rssl->ssl, rv);
if ((c->flags & REDIS_BLOCK) == 0 && maybeCheckWant(rssl, err)) {
return 0;
} else {
__redisSetError(c, REDIS_ERR_IO, NULL);
return -1;
}
}
return rv;
}
static void redisSSLAsyncRead(redisAsyncContext *ac) {
int rv;
redisSSLContext *rssl = ac->c.privdata;
redisContext *c = &ac->c;
rssl->wantRead = 0;
if (rssl->pendingWrite) {
int done;
/* This is probably just a write event */
rssl->pendingWrite = 0;
rv = redisBufferWrite(c, &done);
if (rv == REDIS_ERR) {
__redisAsyncDisconnect(ac);
return;
} else if (!done) {
_EL_ADD_WRITE(ac);
}
}
rv = redisBufferRead(c);
if (rv == REDIS_ERR) {
__redisAsyncDisconnect(ac);
} else {
_EL_ADD_READ(ac);
redisProcessCallbacks(ac);
}
}
static void redisSSLAsyncWrite(redisAsyncContext *ac) {
int rv, done = 0;
redisSSLContext *rssl = ac->c.privdata;
redisContext *c = &ac->c;
rssl->pendingWrite = 0;
rv = redisBufferWrite(c, &done);
if (rv == REDIS_ERR) {
__redisAsyncDisconnect(ac);
return;
}
if (!done) {
if (rssl->wantRead) {
/* Need to read-before-write */
rssl->pendingWrite = 1;
_EL_DEL_WRITE(ac);
} else {
/* No extra reads needed, just need to write more */
_EL_ADD_WRITE(ac);
}
} else {
/* Already done! */
_EL_DEL_WRITE(ac);
}
/* Always reschedule a read */
_EL_ADD_READ(ac);
}
redisContextFuncs redisContextSSLFuncs = {
.free_privdata = redisSSLFreeContext,
.async_read = redisSSLAsyncRead,
.async_write = redisSSLAsyncWrite,
.read = redisSSLRead,
.write = redisSSLWrite
};
+83 -10
View File
@@ -13,12 +13,16 @@
#include <limits.h>
#include "hiredis.h"
#ifdef HIREDIS_TEST_SSL
#include "hiredis_ssl.h"
#endif
#include "net.h"
enum connection_type {
CONN_TCP,
CONN_UNIX,
CONN_FD
CONN_FD,
CONN_SSL
};
struct config {
@@ -33,6 +37,14 @@ struct config {
struct {
const char *path;
} unix_sock;
struct {
const char *host;
int port;
const char *ca_cert;
const char *cert;
const char *key;
} ssl;
};
/* The following lines make up our testing "framework" :) */
@@ -93,11 +105,27 @@ static int disconnect(redisContext *c, int keep_fd) {
return -1;
}
static void do_ssl_handshake(redisContext *c, struct config config) {
#ifdef HIREDIS_TEST_SSL
redisSecureConnection(c, config.ssl.ca_cert, config.ssl.cert, config.ssl.key, NULL);
if (c->err) {
printf("SSL error: %s\n", c->errstr);
redisFree(c);
exit(1);
}
#else
(void) c;
(void) config;
#endif
}
static redisContext *do_connect(struct config config) {
redisContext *c = NULL;
if (config.type == CONN_TCP) {
c = redisConnect(config.tcp.host, config.tcp.port);
} else if (config.type == CONN_SSL) {
c = redisConnect(config.ssl.host, config.ssl.port);
} else if (config.type == CONN_UNIX) {
c = redisConnectUnix(config.unix_sock.path);
} else if (config.type == CONN_FD) {
@@ -121,9 +149,21 @@ static redisContext *do_connect(struct config config) {
exit(1);
}
if (config.type == CONN_SSL) {
do_ssl_handshake(c, config);
}
return select_database(c);
}
static void do_reconnect(redisContext *c, struct config config) {
redisReconnect(c);
if (config.type == CONN_SSL) {
do_ssl_handshake(c, config);
}
}
static void test_format_commands(void) {
char *cmd;
int len;
@@ -360,7 +400,8 @@ static void test_reply_reader(void) {
freeReplyObject(reply);
redisReaderFree(reader);
test("Set error when array > INT_MAX: ");
#if LLONG_MAX > SIZE_MAX
test("Set error when array > SIZE_MAX: ");
reader = redisReaderCreate();
redisReaderFeed(reader, "*9223372036854775807\r\n+asdf\r\n",29);
ret = redisReaderGetReply(reader,&reply);
@@ -369,7 +410,6 @@ static void test_reply_reader(void) {
freeReplyObject(reply);
redisReaderFree(reader);
#if LLONG_MAX > SIZE_MAX
test("Set error when bulk > SIZE_MAX: ");
reader = redisReaderCreate();
redisReaderFeed(reader, "$9223372036854775807\r\nasdf\r\n",28);
@@ -434,22 +474,23 @@ static void test_free_null(void) {
test_cond(reply == NULL);
}
#define HIREDIS_BAD_DOMAIN "idontexist-noreally.com"
static void test_blocking_connection_errors(void) {
redisContext *c;
struct addrinfo hints = {.ai_family = AF_INET};
struct addrinfo *ai_tmp = NULL;
const char *bad_domain = "idontexist.com";
int rv = getaddrinfo(bad_domain, "6379", &hints, &ai_tmp);
int rv = getaddrinfo(HIREDIS_BAD_DOMAIN, "6379", &hints, &ai_tmp);
if (rv != 0) {
// Address does *not* exist
test("Returns error when host cannot be resolved: ");
// First see if this domain name *actually* resolves to NXDOMAIN
c = redisConnect("dontexist.com", 6379);
c = redisConnect(HIREDIS_BAD_DOMAIN, 6379);
test_cond(
c->err == REDIS_ERR_OTHER &&
(strcmp(c->errstr, "Name or service not known") == 0 ||
strcmp(c->errstr, "Can't resolve: sadkfjaskfjsa.com") == 0 ||
strcmp(c->errstr, "Can't resolve: " HIREDIS_BAD_DOMAIN) == 0 ||
strcmp(c->errstr, "Name does not resolve") == 0 ||
strcmp(c->errstr,
"nodename nor servname provided, or not known") == 0 ||
strcmp(c->errstr, "No address associated with hostname") == 0 ||
@@ -574,7 +615,8 @@ static void test_blocking_connection_timeouts(struct config config) {
c = do_connect(config);
test("Does not return a reply when the command times out: ");
s = write(c->fd, cmd, strlen(cmd));
redisAppendFormattedCommand(c, cmd, strlen(cmd));
s = c->funcs->write(c);
tv.tv_sec = 0;
tv.tv_usec = 10000;
redisSetTimeout(c, tv);
@@ -583,7 +625,7 @@ static void test_blocking_connection_timeouts(struct config config) {
freeReplyObject(reply);
test("Reconnect properly reconnects after a timeout: ");
redisReconnect(c);
do_reconnect(c, config);
reply = redisCommand(c, "PING");
test_cond(reply != NULL && reply->type == REDIS_REPLY_STATUS && strcmp(reply->str, "PONG") == 0);
freeReplyObject(reply);
@@ -591,7 +633,7 @@ static void test_blocking_connection_timeouts(struct config config) {
test("Reconnect properly uses owned parameters: ");
config.tcp.host = "foo";
config.unix_sock.path = "foo";
redisReconnect(c);
do_reconnect(c, config);
reply = redisCommand(c, "PING");
test_cond(reply != NULL && reply->type == REDIS_REPLY_STATUS && strcmp(reply->str, "PONG") == 0);
freeReplyObject(reply);
@@ -894,6 +936,23 @@ int main(int argc, char **argv) {
throughput = 0;
} else if (argc >= 1 && !strcmp(argv[0],"--skip-inherit-fd")) {
test_inherit_fd = 0;
#ifdef HIREDIS_TEST_SSL
} else if (argc >= 2 && !strcmp(argv[0],"--ssl-port")) {
argv++; argc--;
cfg.ssl.port = atoi(argv[0]);
} else if (argc >= 2 && !strcmp(argv[0],"--ssl-host")) {
argv++; argc--;
cfg.ssl.host = argv[0];
} else if (argc >= 2 && !strcmp(argv[0],"--ssl-ca-cert")) {
argv++; argc--;
cfg.ssl.ca_cert = argv[0];
} else if (argc >= 2 && !strcmp(argv[0],"--ssl-cert")) {
argv++; argc--;
cfg.ssl.cert = argv[0];
} else if (argc >= 2 && !strcmp(argv[0],"--ssl-key")) {
argv++; argc--;
cfg.ssl.key = argv[0];
#endif
} else {
fprintf(stderr, "Invalid argument: %s\n", argv[0]);
exit(1);
@@ -922,6 +981,20 @@ int main(int argc, char **argv) {
test_blocking_io_errors(cfg);
if (throughput) test_throughput(cfg);
#ifdef HIREDIS_TEST_SSL
if (cfg.ssl.port && cfg.ssl.host) {
printf("\nTesting against SSL connection (%s:%d):\n", cfg.ssl.host, cfg.ssl.port);
cfg.type = CONN_SSL;
test_blocking_connection(cfg);
test_blocking_connection_timeouts(cfg);
test_blocking_io_errors(cfg);
test_invalid_timeout_errors(cfg);
test_append_formatted_commands(cfg);
if (throughput) test_throughput(cfg);
}
#endif
if (test_inherit_fd) {
printf("\nTesting against inherited fd (%s):\n", cfg.unix_sock.path);
cfg.type = CONN_FD;
Vendored Executable
+70
View File
@@ -0,0 +1,70 @@
#!/bin/sh -ue
REDIS_SERVER=${REDIS_SERVER:-redis-server}
REDIS_PORT=${REDIS_PORT:-56379}
REDIS_SSL_PORT=${REDIS_SSL_PORT:-56443}
TEST_SSL=${TEST_SSL:-0}
SSL_TEST_ARGS=
tmpdir=$(mktemp -d)
PID_FILE=${tmpdir}/hiredis-test-redis.pid
SOCK_FILE=${tmpdir}/hiredis-test-redis.sock
if [ "$TEST_SSL" = "1" ]; then
SSL_CA_CERT=${tmpdir}/ca.crt
SSL_CA_KEY=${tmpdir}/ca.key
SSL_CERT=${tmpdir}/redis.crt
SSL_KEY=${tmpdir}/redis.key
openssl genrsa -out ${tmpdir}/ca.key 4096
openssl req \
-x509 -new -nodes -sha256 \
-key ${SSL_CA_KEY} \
-days 3650 \
-subj '/CN=Hiredis Test CA' \
-out ${SSL_CA_CERT}
openssl genrsa -out ${SSL_KEY} 2048
openssl req \
-new -sha256 \
-key ${SSL_KEY} \
-subj '/CN=Hiredis Test Cert' | \
openssl x509 \
-req -sha256 \
-CA ${SSL_CA_CERT} \
-CAkey ${SSL_CA_KEY} \
-CAserial ${tmpdir}/ca.txt \
-CAcreateserial \
-days 365 \
-out ${SSL_CERT}
SSL_TEST_ARGS="--ssl-host 127.0.0.1 --ssl-port ${REDIS_SSL_PORT} --ssl-ca-cert ${SSL_CA_CERT} --ssl-cert ${SSL_CERT} --ssl-key ${SSL_KEY}"
fi
cleanup() {
set +e
kill $(cat ${PID_FILE})
rm -rf ${tmpdir}
}
trap cleanup INT TERM EXIT
cat > ${tmpdir}/redis.conf <<EOF
daemonize yes
pidfile ${PID_FILE}
port ${REDIS_PORT}
bind 127.0.0.1
unixsocket ${SOCK_FILE}
EOF
if [ "$TEST_SSL" = "1" ]; then
cat >> ${tmpdir}/redis.conf <<EOF
tls-port ${REDIS_SSL_PORT}
tls-ca-cert-file ${SSL_CA_CERT}
tls-cert-file ${SSL_CERT}
tls-key-file ${SSL_KEY}
EOF
fi
cat ${tmpdir}/redis.conf
${REDIS_SERVER} ${tmpdir}/redis.conf
${TEST_PREFIX:-} ./hiredis-test -h 127.0.0.1 -p ${REDIS_PORT} -s ${SOCK_FILE} ${SSL_TEST_ARGS}
+16 -2
View File
@@ -2,10 +2,20 @@
#define _WIN32_HELPER_INCLUDE
#ifdef _MSC_VER
#include <winsock2.h> /* for struct timeval */
#ifndef inline
#define inline __inline
#endif
#ifndef strcasecmp
#define strcasecmp stricmp
#endif
#ifndef strncasecmp
#define strncasecmp strnicmp
#endif
#ifndef va_copy
#define va_copy(d,s) ((d) = (s))
#endif
@@ -37,6 +47,10 @@ __inline int c99_snprintf(char* str, size_t size, const char* format, ...)
return count;
}
#endif
#endif /* _MSC_VER */
#endif
#endif
#ifdef _WIN32
#define strerror_r(errno,buf,len) strerror_s(buf,len,errno)
#endif /* _WIN32 */
#endif /* _WIN32_HELPER_INCLUDE */
@@ -216,7 +216,7 @@ ixalloc(tsdn_t *tsdn, void *ptr, size_t oldsize, size_t size, size_t extra,
}
JEMALLOC_ALWAYS_INLINE int
iget_defrag_hint(tsdn_t *tsdn, void* ptr, int *bin_util, int *run_util) {
iget_defrag_hint(tsdn_t *tsdn, void* ptr) {
int defrag = 0;
rtree_ctx_t rtree_ctx_fallback;
rtree_ctx_t *rtree_ctx = tsdn_rtree_ctx(tsdn, &rtree_ctx_fallback);
@@ -232,11 +232,22 @@ iget_defrag_hint(tsdn_t *tsdn, void* ptr, int *bin_util, int *run_util) {
malloc_mutex_lock(tsdn, &bin->lock);
/* don't bother moving allocations from the slab currently used for new allocations */
if (slab != bin->slabcur) {
const bin_info_t *bin_info = &bin_infos[binind];
size_t availregs = bin_info->nregs * bin->stats.curslabs;
*bin_util = ((long long)bin->stats.curregs<<16) / availregs;
*run_util = ((long long)(bin_info->nregs - extent_nfree_get(slab))<<16) / bin_info->nregs;
defrag = 1;
int free_in_slab = extent_nfree_get(slab);
if (free_in_slab) {
const bin_info_t *bin_info = &bin_infos[binind];
int curslabs = bin->stats.curslabs;
size_t curregs = bin->stats.curregs;
if (bin->slabcur) {
/* remove slabcur from the overall utilization */
curregs -= bin_info->nregs - extent_nfree_get(bin->slabcur);
curslabs -= 1;
}
/* Compare the utilization ratio of the slab in question to the total average,
* to avoid precision lost and division, we do that by extrapolating the usage
* of the slab as if all slabs have the same usage. If this slab is less used
* than the average, we'll prefer to evict the data to hopefully more used ones */
defrag = (bin_info->nregs - free_in_slab) * curslabs <= curregs;
}
}
malloc_mutex_unlock(tsdn, &bin->lock);
}
+7 -1
View File
@@ -787,7 +787,13 @@ background_thread_stats_read(tsdn_t *tsdn, background_thread_stats_t *stats) {
nstime_init(&stats->run_interval, 0);
for (unsigned i = 0; i < max_background_threads; i++) {
background_thread_info_t *info = &background_thread_info[i];
malloc_mutex_lock(tsdn, &info->mtx);
if (malloc_mutex_trylock(tsdn, &info->mtx)) {
/*
* Each background thread run may take a long time;
* avoid waiting on the stats if the thread is active.
*/
continue;
}
if (info->state != background_thread_stopped) {
num_runs += info->tot_n_runs;
nstime_add(&stats->run_interval, &info->tot_sleep_time);
+3 -5
View File
@@ -3326,12 +3326,10 @@ jemalloc_postfork_child(void) {
/******************************************************************************/
/* Helps the application decide if a pointer is worth re-allocating in order to reduce fragmentation.
* returns 0 if the allocation is in the currently active run,
* or when it is not causing any frag issue (large or huge bin)
* returns the bin utilization and run utilization both in fixed point 16:16.
* returns 1 if the allocation should be moved, and 0 if the allocation be kept.
* If the application decides to re-allocate it should use MALLOCX_TCACHE_NONE when doing so. */
JEMALLOC_EXPORT int JEMALLOC_NOTHROW
get_defrag_hint(void* ptr, int *bin_util, int *run_util) {
get_defrag_hint(void* ptr) {
assert(ptr != NULL);
return iget_defrag_hint(TSDN_NULL, ptr, bin_util, run_util);
return iget_defrag_hint(TSDN_NULL, ptr);
}
+24 -1
View File
@@ -21,7 +21,7 @@ So what usually happens is either:
The result is a pollution of binaries without line editing support.
So I spent more or less two hours doing a reality check resulting in this little library: is it *really* needed for a line editing library to be 20k lines of code? Apparently not, it is possibe to get a very small, zero configuration, trivial to embed library, that solves the problem. Smaller programs will just include this, supporing line editing out of the box. Larger programs may use this little library or just checking with configure if readline/libedit is available and resorting to Linenoise if not.
So I spent more or less two hours doing a reality check resulting in this little library: is it *really* needed for a line editing library to be 20k lines of code? Apparently not, it is possibe to get a very small, zero configuration, trivial to embed library, that solves the problem. Smaller programs will just include this, supporting line editing out of the box. Larger programs may use this little library or just checking with configure if readline/libedit is available and resorting to Linenoise if not.
## Terminals, in 2010.
@@ -126,6 +126,24 @@ Linenoise has direct support for persisting the history into an history
file. The functions `linenoiseHistorySave` and `linenoiseHistoryLoad` do
just that. Both functions return -1 on error and 0 on success.
## Mask mode
Sometimes it is useful to allow the user to type passwords or other
secrets that should not be displayed. For such situations linenoise supports
a "mask mode" that will just replace the characters the user is typing
with `*` characters, like in the following example:
$ ./linenoise_example
hello> get mykey
echo: 'get mykey'
hello> /mask
hello> *********
You can enable and disable mask mode using the following two functions:
void linenoiseMaskModeEnable(void);
void linenoiseMaskModeDisable(void);
## Completion
Linenoise supports completion, which is the ability to complete the user
@@ -222,3 +240,8 @@ Sometimes you may want to clear the screen as a result of something the
user typed. You can do this by calling the following function:
void linenoiseClearScreen(void);
## Related projects
* [Linenoise NG](https://github.com/arangodb/linenoise-ng) is a fork of Linenoise that aims to add more advanced features like UTF-8 support, Windows support and other features. Uses C++ instead of C as development language.
* [Linenoise-swift](https://github.com/andybest/linenoise-swift) is a reimplementation of Linenoise written in Swift.
+5
View File
@@ -55,6 +55,7 @@ int main(int argc, char **argv) {
*
* The typed string is returned as a malloc() allocated string by
* linenoise, so the user needs to free() it. */
while((line = linenoise("hello> ")) != NULL) {
/* Do something with the string. */
if (line[0] != '\0' && line[0] != '/') {
@@ -65,6 +66,10 @@ int main(int argc, char **argv) {
/* The "/historylen" command will change the history len. */
int len = atoi(line+11);
linenoiseHistorySetMaxLen(len);
} else if (!strncmp(line, "/mask", 5)) {
linenoiseMaskModeEnable();
} else if (!strncmp(line, "/unmask", 7)) {
linenoiseMaskModeDisable();
} else if (line[0] == '/') {
printf("Unreconized command: %s\n", line);
}
+29 -3
View File
@@ -125,6 +125,7 @@ static linenoiseHintsCallback *hintsCallback = NULL;
static linenoiseFreeHintsCallback *freeHintsCallback = NULL;
static struct termios orig_termios; /* In order to restore at exit.*/
static int maskmode = 0; /* Show "***" instead of input. For passwords. */
static int rawmode = 0; /* For atexit() function to check if restore is needed*/
static int mlmode = 0; /* Multi line mode. Default is single line. */
static int atexit_registered = 0; /* Register atexit just 1 time. */
@@ -197,6 +198,19 @@ FILE *lndebug_fp = NULL;
/* ======================= Low level terminal handling ====================== */
/* Enable "mask mode". When it is enabled, instead of the input that
* the user is typing, the terminal will just display a corresponding
* number of asterisks, like "****". This is useful for passwords and other
* secrets that should not be displayed. */
void linenoiseMaskModeEnable(void) {
maskmode = 1;
}
/* Disable mask mode. */
void linenoiseMaskModeDisable(void) {
maskmode = 0;
}
/* Set if to use or not the multi line mode. */
void linenoiseSetMultiLine(int ml) {
mlmode = ml;
@@ -485,6 +499,8 @@ void refreshShowHints(struct abuf *ab, struct linenoiseState *l, int plen) {
if (bold == 1 && color == -1) color = 37;
if (color != -1 || bold != 0)
snprintf(seq,64,"\033[%d;%d;49m",bold,color);
else
seq[0] = '\0';
abAppend(ab,seq,strlen(seq));
abAppend(ab,hint,hintlen);
if (color != -1 || bold != 0)
@@ -523,7 +539,11 @@ static void refreshSingleLine(struct linenoiseState *l) {
abAppend(&ab,seq,strlen(seq));
/* Write the prompt and the current buffer content */
abAppend(&ab,l->prompt,strlen(l->prompt));
abAppend(&ab,buf,len);
if (maskmode == 1) {
while (len--) abAppend(&ab,"*",1);
} else {
abAppend(&ab,buf,len);
}
/* Show hits if any. */
refreshShowHints(&ab,l,plen);
/* Erase to right */
@@ -577,7 +597,12 @@ static void refreshMultiLine(struct linenoiseState *l) {
/* Write the prompt and the current buffer content */
abAppend(&ab,l->prompt,strlen(l->prompt));
abAppend(&ab,l->buf,l->len);
if (maskmode == 1) {
unsigned int i;
for (i = 0; i < l->len; i++) abAppend(&ab,"*",1);
} else {
abAppend(&ab,l->buf,l->len);
}
/* Show hits if any. */
refreshShowHints(&ab,l,plen);
@@ -645,7 +670,8 @@ int linenoiseEditInsert(struct linenoiseState *l, char c) {
if ((!mlmode && l->plen+l->len < l->cols && !hintsCallback)) {
/* Avoid a full update of the line in the
* trivial case. */
if (write(l->ofd,&c,1) == -1) return -1;
char d = (maskmode==1) ? '*' : c;
if (write(l->ofd,&d,1) == -1) return -1;
} else {
refreshLine(l);
}
+2
View File
@@ -65,6 +65,8 @@ int linenoiseHistoryLoad(const char *filename);
void linenoiseClearScreen(void);
void linenoiseSetMultiLine(int ml);
void linenoisePrintKeyCodes(void);
void linenoiseMaskModeEnable(void);
void linenoiseMaskModeDisable(void);
#ifdef __cplusplus
}
+1 -1
View File
@@ -274,7 +274,7 @@ int luaD_precall (lua_State *L, StkId func, int nresults) {
CallInfo *ci;
StkId st, base;
Proto *p = cl->p;
luaD_checkstack(L, p->maxstacksize);
luaD_checkstack(L, p->maxstacksize + p->numparams);
func = restorestack(L, funcr);
if (!p->is_vararg) { /* no varargs? */
base = func + 1;
+6 -4
View File
@@ -89,12 +89,14 @@ typedef struct Header {
} Header;
static int getnum (const char **fmt, int df) {
static int getnum (lua_State *L, const char **fmt, int df) {
if (!isdigit(**fmt)) /* no number? */
return df; /* return default value */
else {
int a = 0;
do {
if (a > (INT_MAX / 10) || a * 10 > (INT_MAX - (**fmt - '0')))
luaL_error(L, "integral size overflow");
a = a*10 + *((*fmt)++) - '0';
} while (isdigit(**fmt));
return a;
@@ -115,9 +117,9 @@ static size_t optsize (lua_State *L, char opt, const char **fmt) {
case 'f': return sizeof(float);
case 'd': return sizeof(double);
case 'x': return 1;
case 'c': return getnum(fmt, 1);
case 'c': return getnum(L, fmt, 1);
case 'i': case 'I': {
int sz = getnum(fmt, sizeof(int));
int sz = getnum(L, fmt, sizeof(int));
if (sz > MAXINTSIZE)
luaL_error(L, "integral size %d is larger than limit of %d",
sz, MAXINTSIZE);
@@ -150,7 +152,7 @@ static void controloptions (lua_State *L, int opt, const char **fmt,
case '>': h->endian = BIG; return;
case '<': h->endian = LITTLE; return;
case '!': {
int a = getnum(fmt, MAXALIGN);
int a = getnum(L, fmt, MAXALIGN);
if (!isp2(a))
luaL_error(L, "alignment %d is not a power of 2", a);
h->align = a;
+397 -56
View File
@@ -129,6 +129,95 @@ timeout 0
# KeyDB default starting with Redis 3.2.1.
tcp-keepalive 300
################################# TLS/SSL #####################################
# By default, TLS/SSL is disabled. To enable it, the "tls-port" configuration
# directive can be used to define TLS-listening ports. To enable TLS on the
# default port, use:
#
# port 0
# tls-port 6379
# Configure a X.509 certificate and private key to use for authenticating the
# server to connected clients, masters or cluster peers. These files should be
# PEM formatted.
#
# tls-cert-file redis.crt
# tls-key-file redis.key
# Configure a DH parameters file to enable Diffie-Hellman (DH) key exchange:
#
# tls-dh-params-file redis.dh
# Configure a CA certificate(s) bundle or directory to authenticate TLS/SSL
# clients and peers. Redis requires an explicit configuration of at least one
# of these, and will not implicitly use the system wide configuration.
#
# tls-ca-cert-file ca.crt
# tls-ca-cert-dir /etc/ssl/certs
# By default, clients (including replica servers) on a TLS port are required
# to authenticate using valid client side certificates.
#
# If "no" is specified, client certificates are not required and not accepted.
# If "optional" is specified, client certificates are accepted and must be
# valid if provided, but are not required.
#
# tls-auth-clients no
# tls-auth-clients optional
# By default, a Redis replica does not attempt to establish a TLS connection
# with its master.
#
# Use the following directive to enable TLS on replication links.
#
# tls-replication yes
# By default, the Redis Cluster bus uses a plain TCP connection. To enable
# TLS for the bus protocol, use the following directive:
#
# tls-cluster yes
# Explicitly specify TLS versions to support. Allowed values are case insensitive
# and include "TLSv1", "TLSv1.1", "TLSv1.2", "TLSv1.3" (OpenSSL >= 1.1.1) or
# any combination. To enable only TLSv1.2 and TLSv1.3, use:
#
# tls-protocols "TLSv1.2 TLSv1.3"
# Configure allowed ciphers. See the ciphers(1ssl) manpage for more information
# about the syntax of this string.
#
# Note: this configuration applies only to <= TLSv1.2.
#
# tls-ciphers DEFAULT:!MEDIUM
# Configure allowed TLSv1.3 ciphersuites. See the ciphers(1ssl) manpage for more
# information about the syntax of this string, and specifically for TLSv1.3
# ciphersuites.
#
# tls-ciphersuites TLS_CHACHA20_POLY1305_SHA256
# When choosing a cipher, use the server's preference instead of the client
# preference. By default, the server follows the client's preference.
#
# tls-prefer-server-ciphers yes
# By default, TLS session caching is enabled to allow faster and less expensive
# reconnections by clients that support it. Use the following directive to disable
# caching.
#
# tls-session-caching no
# Change the default number of TLS sessions cached. A zero value sets the cache
# to unlimited size. The default size is 20480.
#
# tls-session-cache-size 5000
# Change the default timeout of cached TLS sessions. The default timeout is 300
# seconds.
#
# tls-session-cache-timeout 60
################################# GENERAL #####################################
# By default KeyDB does not run as a daemon. Use 'yes' if you need it.
@@ -252,6 +341,19 @@ rdbchecksum yes
# The filename where to dump the DB
dbfilename dump.rdb
# Remove RDB files used by replication in instances without persistence
# enabled. By default this option is disabled, however there are environments
# where for regulations or other security concerns, RDB files persisted on
# disk by masters in order to feed replicas, or stored on disk by replicas
# in order to load them for the initial synchronization, should be deleted
# ASAP. Note that this option ONLY WORKS in instances that have both AOF
# and RDB persistence disabled, otherwise is completely ignored.
#
# An alternative (and sometimes better) way to obtain the same effect is
# to use diskless replication on both master and replicas instances. However
# in the case of replicas, diskless is not always an option.
rdb-del-sync-files no
# The working directory.
#
# The DB will be written inside this directory, with the filename specified
@@ -294,9 +396,9 @@ dir ./
#
# However this is not enough if you are using KeyDB ACLs (for Redis version
# 6 or greater), and the default user is not capable of running the PSYNC
# command and/or other commands needed for replication. In this case it's
# better to configure a special user to use with replication, and specify the
# masteruser configuration as such:
# command and/or other commands needed for replication (gathered in the
# @replication group). In this case it's better to configure a special user to
# use with replication, and specify the masteruser configuration as such:
#
# masteruser <username>
#
@@ -318,6 +420,20 @@ dir ./
#
replica-serve-stale-data yes
# Active Replicas will allow read only data access while loading remote RDBs
# provided they are permitted to serve stale data. As an option you may also
# permit them to accept write commands. This is an EXPERIMENTAL feature and
# may result in commands not being fully synchronized
#
# allow-write-during-load no
# You can modify the number of masters necessary to form a replica quorum when
# multi-master is enabled and replica-serve-stale-data is "no". By default
# this is set to -1 which implies the number of known masters (e.g. those
# you added with replicaof)
#
# replica-quorum -1
# You can configure a replica instance to accept writes or not. Writing against
# a replica instance may be useful to store some ephemeral data (because data
# written on a replica will be easily deleted after resync with the master) but
@@ -336,13 +452,11 @@ replica-read-only yes
# Replication SYNC strategy: disk or socket.
#
# -------------------------------------------------------
# WARNING: DISKLESS REPLICATION IS EXPERIMENTAL CURRENTLY
# -------------------------------------------------------
# New replicas and reconnecting replicas that are not able to continue the
# replication process just receiving differences, need to do what is called a
# "full synchronization". An RDB file is transmitted from the master to the
# replicas.
#
# New replicas and reconnecting replicas that are not able to continue the replication
# process just receiving differences, need to do what is called a "full
# synchronization". An RDB file is transmitted from the master to the replicas.
# The transmission can happen in two different ways:
#
# 1) Disk-backed: The KeyDB master creates a new process that writes the RDB
@@ -352,14 +466,14 @@ replica-read-only yes
# RDB file to replica sockets, without touching the disk at all.
#
# With disk-backed replication, while the RDB file is generated, more replicas
# can be queued and served with the RDB file as soon as the current child producing
# the RDB file finishes its work. With diskless replication instead once
# the transfer starts, new replicas arriving will be queued and a new transfer
# will start when the current one terminates.
# can be queued and served with the RDB file as soon as the current child
# producing the RDB file finishes its work. With diskless replication instead
# once the transfer starts, new replicas arriving will be queued and a new
# transfer will start when the current one terminates.
#
# When diskless replication is used, the master waits a configurable amount of
# time (in seconds) before starting the transfer in the hope that multiple replicas
# will arrive and the transfer can be parallelized.
# time (in seconds) before starting the transfer in the hope that multiple
# replicas will arrive and the transfer can be parallelized.
#
# With slow disks and fast (large bandwidth) networks, diskless replication
# works better.
@@ -370,16 +484,42 @@ repl-diskless-sync no
# to the replicas.
#
# This is important since once the transfer starts, it is not possible to serve
# new replicas arriving, that will be queued for the next RDB transfer, so the server
# waits a delay in order to let more replicas arrive.
# new replicas arriving, that will be queued for the next RDB transfer, so the
# server waits a delay in order to let more replicas arrive.
#
# The delay is specified in seconds, and by default is 5 seconds. To disable
# it entirely just set it to 0 seconds and the transfer will start ASAP.
repl-diskless-sync-delay 5
# Replicas send PINGs to server in a predefined interval. It's possible to change
# this interval with the repl_ping_replica_period option. The default value is 10
# seconds.
# -----------------------------------------------------------------------------
# WARNING: RDB diskless load is experimental. Since in this setup the replica
# does not immediately store an RDB on disk, it may cause data loss during
# failovers. RDB diskless load + Redis modules not handling I/O reads may also
# cause Redis to abort in case of I/O errors during the initial synchronization
# stage with the master. Use only if your do what you are doing.
# -----------------------------------------------------------------------------
#
# Replica can load the RDB it reads from the replication link directly from the
# socket, or store the RDB to a file and read that file after it was completely
# recived from the master.
#
# In many cases the disk is slower than the network, and storing and loading
# the RDB file may increase replication time (and even increase the master's
# Copy on Write memory and salve buffers).
# However, parsing the RDB file directly from the socket may mean that we have
# to flush the contents of the current database before the full rdb was
# received. For this reason we have the following options:
#
# "disabled" - Don't use diskless load (store the rdb file to the disk first)
# "on-empty-db" - Use diskless load only when it is completely safe.
# "swapdb" - Keep a copy of the current db contents in RAM while parsing
# the data directly from the socket. note that this requires
# sufficient memory, if you don't have it, you risk an OOM kill.
repl-diskless-load disabled
# Replicas send PINGs to server in a predefined interval. It's possible to
# change this interval with the repl_ping_replica_period option. The default
# value is 10 seconds.
#
# repl-ping-replica-period 10
@@ -411,10 +551,10 @@ repl-diskless-sync-delay 5
repl-disable-tcp-nodelay no
# Set the replication backlog size. The backlog is a buffer that accumulates
# replica data when replicas are disconnected for some time, so that when a replica
# wants to reconnect again, often a full resync is not needed, but a partial
# resync is enough, just passing the portion of data the replica missed while
# disconnected.
# replica data when replicas are disconnected for some time, so that when a
# replica wants to reconnect again, often a full resync is not needed, but a
# partial resync is enough, just passing the portion of data the replica
# missed while disconnected.
#
# The bigger the replication backlog, the longer the time the replica can be
# disconnected and later be able to perform a partial resynchronization.
@@ -436,13 +576,13 @@ repl-disable-tcp-nodelay no
#
# repl-backlog-ttl 3600
# The replica priority is an integer number published by KeyDB in the INFO output.
# It is used by KeyDB Sentinel in order to select a replica to promote into a
# master if the master is no longer working correctly.
# The replica priority is an integer number published by KeyDB in the INFO
# output. It is used by Redis Sentinel in order to select a replica to promote
# into a master if the master is no longer working correctly.
#
# A replica with a low priority number is considered better for promotion, so
# for instance if there are three replicas with priority 10, 100, 25 Sentinel will
# pick the one with priority 10, that is the lowest.
# for instance if there are three replicas with priority 10, 100, 25 Sentinel
# will pick the one with priority 10, that is the lowest.
#
# However a special priority of 0 marks the replica as not able to perform the
# role of master, so a replica with priority of 0 will never be selected by
@@ -502,6 +642,42 @@ replica-priority 100
# replica-announce-ip 5.5.5.5
# replica-announce-port 1234
############################### KEYS TRACKING #################################
# Redis implements server assisted support for client side caching of values.
# This is implemented using an invalidation table that remembers, using
# 16 millions of slots, what clients may have certain subsets of keys. In turn
# this is used in order to send invalidation messages to clients. Please
# to understand more about the feature check this page:
#
# https://redis.io/topics/client-side-caching
#
# When tracking is enabled for a client, all the read only queries are assumed
# to be cached: this will force Redis to store information in the invalidation
# table. When keys are modified, such information is flushed away, and
# invalidation messages are sent to the clients. However if the workload is
# heavily dominated by reads, Redis could use more and more memory in order
# to track the keys fetched by many clients.
#
# For this reason it is possible to configure a maximum fill value for the
# invalidation table. By default it is set to 1M of keys, and once this limit
# is reached, Redis will start to evict keys in the invalidation table
# even if they were not modified, just to reclaim memory: this will in turn
# force the clients to invalidate the cached values. Basically the table
# maximum size is a trade off between the memory you want to spend server
# side to track information about who cached what, and the ability of clients
# to retain cached objects in memory.
#
# If you set the value to 0, it means there are no limits, and Redis will
# retain as many keys as needed in the invalidation table.
# In the "stats" INFO section, you can find information about the number of
# keys in the invalidation table at every given moment.
#
# Note: when key tracking is used in broadcasting mode, no memory is used
# in the server side so this setting is useless.
#
# tracking-table-max-keys 1000000
################################## SECURITY ###################################
# Warning: since KeyDB is pretty fast an outside user can try up to
@@ -598,6 +774,14 @@ replica-priority 100
# For more information about ACL configuration please refer to
# the Redis web site at https://redis.io/topics/acl
# ACL LOG
#
# The ACL Log tracks failed commands and authentication events associated
# with ACLs. The ACL Log is useful to troubleshoot failed commands blocked
# by ACLs. The ACL Log is stored in memory. You can reclaim memory with
# ACL LOG RESET. Define the maximum entry length of the ACL Log below.
acllog-max-len 128
# Using an external ACL file
#
# Instead of configuring users here in this file, it is possible to use
@@ -654,6 +838,11 @@ replica-priority 100
# Once the limit is reached KeyDB will close all the new connections sending
# an error 'max number of clients reached'.
#
# IMPORTANT: When Redis Cluster is used, the max number of connections is also
# shared with the cluster bus: every node in the cluster will use two
# connections, one incoming and another outgoing. It is important to size the
# limit accordingly in case of very large clusters.
#
# maxclients 10000
############################## MEMORY MANAGEMENT ################################
@@ -684,13 +873,13 @@ replica-priority 100
# maxmemory <bytes>
# MAXMEMORY POLICY: how KeyDB will select what to remove when maxmemory
# is reached. You can select among five behaviors:
# is reached. You can select one from the following behaviors:
#
# volatile-lru -> Evict using approximated LRU among the keys with an expire set.
# volatile-lru -> Evict using approximated LRU, only keys with an expire set.
# allkeys-lru -> Evict any key using approximated LRU.
# volatile-lfu -> Evict using approximated LFU among the keys with an expire set.
# volatile-lfu -> Evict using approximated LFU, only keys with an expire set.
# allkeys-lfu -> Evict any key using approximated LFU.
# volatile-random -> Remove a random key among the ones with an expire set.
# volatile-random -> Remove a random key having an expire set.
# allkeys-random -> Remove a random key, any key.
# volatile-ttl -> Remove the key with the nearest expire time (minor TTL)
# noeviction -> Don't evict anything, just return an error on write operations.
@@ -731,20 +920,37 @@ replica-priority 100
# DEL commands to the replica as keys evict in the master side.
#
# This behavior ensures that masters and replicas stay consistent, and is usually
# what you want, however if your replica is writable, or you want the replica to have
# a different memory setting, and you are sure all the writes performed to the
# replica are idempotent, then you may change this default (but be sure to understand
# what you are doing).
# what you want, however if your replica is writable, or you want the replica
# to have a different memory setting, and you are sure all the writes performed
# to the replica are idempotent, then you may change this default (but be sure
# to understand what you are doing).
#
# Note that since the replica by default does not evict, it may end using more
# memory than the one set via maxmemory (there are certain buffers that may
# be larger on the replica, or data structures may sometimes take more memory and so
# forth). So make sure you monitor your replicas and make sure they have enough
# memory to never hit a real out-of-memory condition before the master hits
# the configured maxmemory setting.
# be larger on the replica, or data structures may sometimes take more memory
# and so forth). So make sure you monitor your replicas and make sure they
# have enough memory to never hit a real out-of-memory condition before the
# master hits the configured maxmemory setting.
#
# replica-ignore-maxmemory yes
# Redis reclaims expired keys in two ways: upon access when those keys are
# found to be expired, and also in background, in what is called the
# "active expire key". The key space is slowly and interactively scanned
# looking for expired keys to reclaim, so that it is possible to free memory
# of keys that are expired and will never be accessed again in a short time.
#
# The default effort of the expire cycle will try to avoid having more than
# ten percent of expired keys still in memory, and will try to avoid consuming
# more than 25% of total memory and to add latency to the system. However
# it is possible to increase the expire "effort" that is normally set to
# "1", to a greater value, up to the value "10". At its maximum value the
# system will use more CPU, longer cycles (and technically may introduce
# more latency), and will tollerate less already expired keys still present
# in the system. It's a tradeoff betweeen memory, CPU and latecy.
#
# active-expire-effort 1
############################# LAZY FREEING ####################################
# KeyDB has two primitives to delete keys. One is called DEL and is a blocking
@@ -787,13 +993,92 @@ replica-priority 100
# In all the above cases the default is to delete objects in a blocking way,
# like if DEL was called. However you can configure each case specifically
# in order to instead release memory in a non-blocking way like if UNLINK
# was called, using the following configuration directives:
# was called, using the following configuration directives.
lazyfree-lazy-eviction no
lazyfree-lazy-expire no
lazyfree-lazy-server-del no
replica-lazy-flush no
# It is also possible, for the case when to replace the user code DEL calls
# with UNLINK calls is not easy, to modify the default behavior of the DEL
# command to act exactly like UNLINK, using the following configuration
# directive:
lazyfree-lazy-user-del no
################################ THREADED I/O #################################
# Redis is mostly single threaded, however there are certain threaded
# operations such as UNLINK, slow I/O accesses and other things that are
# performed on side threads.
#
# Now it is also possible to handle Redis clients socket reads and writes
# in different I/O threads. Since especially writing is so slow, normally
# Redis users use pipelining in order to speedup the Redis performances per
# core, and spawn multiple instances in order to scale more. Using I/O
# threads it is possible to easily speedup two times Redis without resorting
# to pipelining nor sharding of the instance.
#
# By default threading is disabled, we suggest enabling it only in machines
# that have at least 4 or more cores, leaving at least one spare core.
# Using more than 8 threads is unlikely to help much. We also recommend using
# threaded I/O only if you actually have performance problems, with Redis
# instances being able to use a quite big percentage of CPU time, otherwise
# there is no point in using this feature.
#
# So for instance if you have a four cores boxes, try to use 2 or 3 I/O
# threads, if you have a 8 cores, try to use 6 threads. In order to
# enable I/O threads use the following configuration directive:
#
# io-threads 4
#
# Setting io-threads to 1 will just use the main thread as usually.
# When I/O threads are enabled, we only use threads for writes, that is
# to thread the write(2) syscall and transfer the client buffers to the
# socket. However it is also possible to enable threading of reads and
# protocol parsing using the following configuration directive, by setting
# it to yes:
#
# io-threads-do-reads no
#
# Usually threading reads doesn't help much.
#
# NOTE 1: This configuration directive cannot be changed at runtime via
# CONFIG SET. Aso this feature currently does not work when SSL is
# enabled.
#
# NOTE 2: If you want to test the Redis speedup using redis-benchmark, make
# sure you also run the benchmark itself in threaded mode, using the
# --threads option to match the number of Redis theads, otherwise you'll not
# be able to notice the improvements.
############################ KERNEL OOM CONTROL ##############################
# On Linux, it is possible to hint the kernel OOM killer on what processes
# should be killed first when out of memory.
#
# Enabling this feature makes Redis actively control the oom_score_adj value
# for all its processes, depending on their role. The default scores will
# attempt to have background child processes killed before all others, and
# replicas killed before masters.
oom-score-adj no
# When oom-score-adj is used, this directive controls the specific values used
# for master, replica and background child processes. Values range -1000 to
# 1000 (higher means more likely to be killed).
#
# Unprivileged processes (not root, and without CAP_SYS_RESOURCE capabilities)
# can freely increase their value, but not decrease it below its initial
# settings.
#
# Values are used relative to the initial value of oom_score_adj when the server
# starts. Because typically the initial value is 0, they will often match the
# absolute values.
oom-score-adj-values 0 200 800
############################## APPEND ONLY MODE ###############################
# By default KeyDB asynchronously dumps the dataset on disk. This mode is
@@ -1050,6 +1335,22 @@ lua-time-limit 5000
#
# cluster-replica-no-failover no
# This option, when set to yes, allows nodes to serve read traffic while the
# the cluster is in a down state, as long as it believes it owns the slots.
#
# This is useful for two cases. The first case is for when an application
# doesn't require consistency of data during node failures or network partitions.
# One example of this is a cache, where as long as the node has the data it
# should be able to serve it.
#
# The second use case is for configurations that don't meet the recommended
# three shards but want to enable cluster mode and scale later. A
# master outage in a 1 or 2 shard configuration causes a read/write outage to the
# entire cluster without this option set, with it set there is only a write outage.
# Without a quorum of masters, slot ownership will not change automatically.
#
# cluster-allow-reads-when-down no
# In order to setup your cluster make sure to read the documentation
# available at http://redis.io web site.
@@ -1156,7 +1457,11 @@ latency-monitor-threshold 0
# z Sorted set commands
# x Expired events (events generated every time a key expires)
# e Evicted events (events generated when a key is evicted for maxmemory)
# A Alias for g$lshzxe, so that the "AKE" string means all the events.
# t Stream commands
# m Key-miss events (Note: It is not included in the 'A' class)
# A Alias for g$lshzxet, so that the "AKE" string means all the events
# (Except key-miss events which are excluded from 'A' due to their
# unique nature).
#
# The "notify-keyspace-events" takes as argument a string that is composed
# of zero or multiple characters. The empty string means that notifications
@@ -1182,7 +1487,7 @@ notify-keyspace-events ""
# KeyDB contains an implementation of the Gopher protocol, as specified in
# the RFC 1436 (https://www.ietf.org/rfc/rfc1436.txt).
#
# The Gopher protocol was very popular in the late '90s. It is an alternative
# The Gopher protocol was very popular in the late '90s. It is an alternative
# to the web, and the implementation both server and client side is so simple
# that the KeyDB server has just 100 lines of code in order to implement this
# support.
@@ -1220,7 +1525,7 @@ notify-keyspace-events ""
# to server Gopher pages MAKE SURE TO SET A PASSWORD to the instance.
# Once a password is set:
#
# 1. The Gopher server (when enabled, not by default) will kill serve
# 1. The Gopher server (when enabled, not by default) will still serve
# content via Gopher.
# 2. However other commands cannot be called before the client will
# authenticate.
@@ -1376,7 +1681,7 @@ client-output-buffer-limit pubsub 32mb 8mb 60
# In the KeyDB protocol, bulk requests, that are, elements representing single
# strings, are normally limited ot 512 mb. However you can change this limit
# here.
# here, but must be 1mb or greater
#
# proto-max-bulk-len 512mb
@@ -1406,7 +1711,7 @@ hz 10
# offers, and enables by default, the ability to use an adaptive HZ value
# which will temporary raise when there are many connected clients.
#
# When dynamic HZ is enabled, the actual configured HZ will be used as
# When dynamic HZ is enabled, the actual configured HZ will be used
# as a baseline, but multiples of the configured HZ value will be actually
# used as needed once more clients are connected. In this way an idle
# instance will use very little CPU time while a busy instance will be
@@ -1479,10 +1784,6 @@ rdb-save-incremental-fsync yes
########################### ACTIVE DEFRAGMENTATION #######################
#
# WARNING THIS FEATURE IS EXPERIMENTAL. However it was stress tested
# even in production and manually tested by multiple engineers for some
# time.
#
# What is active defragmentation?
# -------------------------------
#
@@ -1522,7 +1823,7 @@ rdb-save-incremental-fsync yes
# a good idea to leave the defaults untouched.
# Enabled active defragmentation
# activedefrag yes
# activedefrag no
# Minimum amount of fragmentation waste to start active defrag
# active-defrag-ignore-bytes 100mb
@@ -1533,16 +1834,56 @@ rdb-save-incremental-fsync yes
# Maximum percentage of fragmentation at which we use maximum effort
# active-defrag-threshold-upper 100
# Minimal effort for defrag in CPU percentage
# active-defrag-cycle-min 5
# Minimal effort for defrag in CPU percentage, to be used when the lower
# threshold is reached
# active-defrag-cycle-min 1
# Maximal effort for defrag in CPU percentage
# active-defrag-cycle-max 75
# Maximal effort for defrag in CPU percentage, to be used when the upper
# threshold is reached
# active-defrag-cycle-max 25
# Maximum number of set/hash/zset/list fields that will be processed from
# the main dictionary scan
# active-defrag-max-scan-fields 1000
# Jemalloc background thread for purging will be enabled by default
jemalloc-bg-thread yes
# It is possible to pin different threads and processes of Redis to specific
# CPUs in your system, in order to maximize the performances of the server.
# This is useful both in order to pin different Redis threads in different
# CPUs, but also in order to make sure that multiple Redis instances running
# in the same host will be pinned to different CPUs.
#
# Normally you can do this using the "taskset" command, however it is also
# possible to this via Redis configuration directly, both in Linux and FreeBSD.
#
# You can pin the server/IO threads, bio threads, aof rewrite child process, and
# the bgsave child process. The syntax to specify the cpu list is the same as
# the taskset command:
#
# Set redis server/io threads to cpu affinity 0,2,4,6:
# server_cpulist 0-7:2
#
# Set bio threads to cpu affinity 1,3:
# bio_cpulist 1,3
#
# Set aof rewrite child process to cpu affinity 8,9,10,11:
# aof_rewrite_cpulist 8-11
#
# Set bgsave child process to cpu affinity 1,10,11
# bgsave_cpulist 1,10-11
# The minimum number of clients on a thread before KeyDB assigns new connections to a different thread
# Tuning this parameter is a tradeoff between locking overhead and distributing the workload over multiple cores
# min-clients-per-thread 50
# Avoid forwarding RREPLAY messages to other masters?
# WARNING: This setting is dangerous! You must be certain all masters are connected to each
# other in a true mesh topology or data loss will occur!
# This command can be used to reduce multimaster bus traffic
# multi-master-no-forward no
# Path to directory for file backed scratchpad. The file backed scratchpad
# reduces memory requirements by storing rarely accessed data on disk
# instead of RAM. A temporary file will be created in this directory.
+3
View File
@@ -0,0 +1,3 @@
### KeyDB Packaging
The following directories contain packaging code used to create rpm and deb packages as well as docker images.
+21
View File
@@ -0,0 +1,21 @@
### KeyDB DEB Package Source Builds
This directory contains scripts and components needed to generate debian packages on different distributions/architectures from source
The 'debian' directory contains debian source code for bionic, buster and later distributions as it uses functions only available with debhelper11+. 'debian_dh9' is used for xenial, stretch and earlier distributions using debhelper9.
You will need to install pbuilder `sudo apt install pbuilder` along with other distribution specific dependancies
Generate deb packages with the following script command run from this directory:
```
$ ./deb-buildsource.sh
```
This generates a directory structure, .dsc file, original.tar.gz, and new changelog for the distribution and architecture installed.
When complete the produced debian packages will be located in deb_files_generated directory.
Arguments for the deb-buildsource.sh script that can be passed are either 'None' or '"your custom comments"'. 'None' assumes a permanent changelog entry has been made listing the correct build for the version. If you quote and enter your own comment string, it will be appended to the changelog for that deb package. If no argument is passed a default comment is generated saying that this deb package was generated by this script.
This script has been tested on xenial/bionic/stretch/buster
File diff suppressed because it is too large. Load diff
+244
View File
@@ -0,0 +1,244 @@
# Example sentinel.conf
# *** IMPORTANT ***
#
# By default Sentinel will not be reachable from interfaces different than
# localhost, either use the 'bind' directive to bind to a list of network
# interfaces, or disable protected mode with "protected-mode no" by
# adding it to this configuration file.
#
# Before doing that MAKE SURE the instance is protected from the outside
# world via firewalling or other means.
#
# For example you may use one of the following:
#
# bind 127.0.0.1 192.168.1.1
#
# protected-mode no
# port <sentinel-port>
# The port that this sentinel instance will run on
port 26379
# By default Redis Sentinel does not run as a daemon. Use 'yes' if you need it.
# Note that Redis will write a pid file in /var/run/keydb-sentinel.pid when
# daemonized.
daemonize yes
# When running daemonized, Redis Sentinel writes a pid file in
# /var/run/keydb-sentinel.pid by default. You can specify a custom pid file
# location here.
pidfile /var/run/sentinel/keydb-sentinel.pid
# Specify the log file name. Also the empty string can be used to force
# Sentinel to log on the standard output. Note that if you use standard
# output for logging but daemonize, logs will be sent to /dev/null
logfile /var/log/keydb/keydb-sentinel.log
# sentinel announce-ip <ip>
# sentinel announce-port <port>
#
# The above two configuration directives are useful in environments where,
# because of NAT, Sentinel is reachable from outside via a non-local address.
#
# When announce-ip is provided, the Sentinel will claim the specified IP address
# in HELLO messages used to gossip its presence, instead of auto-detecting the
# local address as it usually does.
#
# Similarly when announce-port is provided and is valid and non-zero, Sentinel
# will announce the specified TCP port.
#
# The two options don't need to be used together, if only announce-ip is
# provided, the Sentinel will announce the specified IP and the server port
# as specified by the "port" option. If only announce-port is provided, the
# Sentinel will announce the auto-detected local IP and the specified port.
#
# Example:
#
# sentinel announce-ip 1.2.3.4
# dir <working-directory>
# Every long running process should have a well-defined working directory.
# For Redis Sentinel to chdir to /tmp at startup is the simplest thing
# for the process to don't interfere with administrative tasks such as
# unmounting filesystems.
dir /var/lib/keydb
# sentinel monitor <master-name> <ip> <keydb-port> <quorum>
#
# Tells Sentinel to monitor this master, and to consider it in O_DOWN
# (Objectively Down) state only if at least <quorum> sentinels agree.
#
# Note that whatever is the ODOWN quorum, a Sentinel will require to
# be elected by the majority of the known Sentinels in order to
# start a failover, so no failover can be performed in minority.
#
# Replicas are auto-discovered, so you don't need to specify replicas in
# any way. Sentinel itself will rewrite this configuration file adding
# the replicas using additional configuration options.
# Also note that the configuration file is rewritten when a
# replica is promoted to master.
#
# Note: master name should not include special characters or spaces.
# The valid charset is A-z 0-9 and the three characters ".-_".
sentinel monitor mymaster 127.0.0.1 6379 2
# sentinel auth-pass <master-name> <password>
#
# Set the password to use to authenticate with the master and replicas.
# Useful if there is a password set in the Redis instances to monitor.
#
# Note that the master password is also used for replicas, so it is not
# possible to set a different password in masters and replicas instances
# if you want to be able to monitor these instances with Sentinel.
#
# However you can have Redis instances without the authentication enabled
# mixed with Redis instances requiring the authentication (as long as the
# password set is the same for all the instances requiring the password) as
# the AUTH command will have no effect in Redis instances with authentication
# switched off.
#
# Example:
#
# sentinel auth-pass mymaster MySUPER--secret-0123passw0rd
# sentinel down-after-milliseconds <master-name> <milliseconds>
#
# Number of milliseconds the master (or any attached replica or sentinel) should
# be unreachable (as in, not acceptable reply to PING, continuously, for the
# specified period) in order to consider it in S_DOWN state (Subjectively
# Down).
#
# Default is 30 seconds.
sentinel down-after-milliseconds mymaster 30000
# sentinel parallel-syncs <master-name> <numreplicas>
#
# How many replicas we can reconfigure to point to the new replica simultaneously
# during the failover. Use a low number if you use the replicas to serve query
# to avoid that all the replicas will be unreachable at about the same
# time while performing the synchronization with the master.
sentinel parallel-syncs mymaster 1
# sentinel failover-timeout <master-name> <milliseconds>
#
# Specifies the failover timeout in milliseconds. It is used in many ways:
#
# - The time needed to re-start a failover after a previous failover was
# already tried against the same master by a given Sentinel, is two
# times the failover timeout.
#
# - The time needed for a replica replicating to a wrong master according
# to a Sentinel current configuration, to be forced to replicate
# with the right master, is exactly the failover timeout (counting since
# the moment a Sentinel detected the misconfiguration).
#
# - The time needed to cancel a failover that is already in progress but
# did not produced any configuration change (SLAVEOF NO ONE yet not
# acknowledged by the promoted replica).
#
# - The maximum time a failover in progress waits for all the replicas to be
# reconfigured as replicas of the new master. However even after this time
# the replicas will be reconfigured by the Sentinels anyway, but not with
# the exact parallel-syncs progression as specified.
#
# Default is 3 minutes.
sentinel failover-timeout mymaster 180000
# SCRIPTS EXECUTION
#
# sentinel notification-script and sentinel reconfig-script are used in order
# to configure scripts that are called to notify the system administrator
# or to reconfigure clients after a failover. The scripts are executed
# with the following rules for error handling:
#
# If script exits with "1" the execution is retried later (up to a maximum
# number of times currently set to 10).
#
# If script exits with "2" (or an higher value) the script execution is
# not retried.
#
# If script terminates because it receives a signal the behavior is the same
# as exit code 1.
#
# A script has a maximum running time of 60 seconds. After this limit is
# reached the script is terminated with a SIGKILL and the execution retried.
# NOTIFICATION SCRIPT
#
# sentinel notification-script <master-name> <script-path>
#
# Call the specified notification script for any sentinel event that is
# generated in the WARNING level (for instance -sdown, -odown, and so forth).
# This script should notify the system administrator via email, SMS, or any
# other messaging system, that there is something wrong with the monitored
# Redis systems.
#
# The script is called with just two arguments: the first is the event type
# and the second the event description.
#
# The script must exist and be executable in order for sentinel to start if
# this option is provided.
#
# Example:
#
# sentinel notification-script mymaster /var/redis/notify.sh
# CLIENTS RECONFIGURATION SCRIPT
#
# sentinel client-reconfig-script <master-name> <script-path>
#
# When the master changed because of a failover a script can be called in
# order to perform application-specific tasks to notify the clients that the
# configuration has changed and the master is at a different address.
#
# The following arguments are passed to the script:
#
# <master-name> <role> <state> <from-ip> <from-port> <to-ip> <to-port>
#
# <state> is currently always "failover"
# <role> is either "leader" or "observer"
#
# The arguments from-ip, from-port, to-ip, to-port are used to communicate
# the old address of the master and the new address of the elected replica
# (now a master).
#
# This script should be resistant to multiple invocations.
#
# Example:
#
# sentinel client-reconfig-script mymaster /var/redis/reconfig.sh
# SECURITY
#
# By default SENTINEL SET will not be able to change the notification-script
# and client-reconfig-script at runtime. This avoids a trivial security issue
# where clients can set the script to anything and trigger a failover in order
# to get the program executed.
sentinel deny-scripts-reconfig yes
# REDIS COMMANDS RENAMING
#
# Sometimes the Redis server has certain commands, that are needed for Sentinel
# to work correctly, renamed to unguessable strings. This is often the case
# of CONFIG and SLAVEOF in the context of providers that provide Redis as
# a service, and don't want the customers to reconfigure the instances outside
# of the administration console.
#
# In such case it is possible to tell Sentinel to use different command names
# instead of the normal ones. For example if the master "mymaster", and the
# associated replicas, have "CONFIG" all renamed to "GUESSME", I could use:
#
# SENTINEL rename-command mymaster CONFIG GUESSME
#
# After such configuration is set, every time Sentinel would use CONFIG it will
# use GUESSME instead. Note that there is no actual need to respect the command
# case, so writing "config guessme" is the same in the example above.
#
# SENTINEL SET can also be used in order to perform this configuration at runtime.
#
# In order to set a command back to its original name (undo the renaming), it
# is possible to just rename a command to itsef:
#
# SENTINEL rename-command mymaster CONFIG CONFIG
+63
View File
@@ -0,0 +1,63 @@
#! /bin/bash
# define parameters used in this deb package build
# enter comments as an argument string quoted. If you do not want comments to be added (ie. you have updated in git repo) type None
if [ $# -eq 0 ]; then
changelog_comments="This is a new source build generated from the deb_build_source.sh script"
else
changelog_comments=$1
fi
echo $changelog_comments
build=1 #change if updated build number required. Default is 1 as convention for keydb is to update minor version
version=$(grep KEYDB_REAL_VERSION ../../src/version.h | awk '{ printf $3 }' | tr -d \")
majorv="${version:0:1}"
distributor=$(lsb_release --id --short)
if [ "$distributor" == "Debian" ]; then
distname=+deb$(lsb_release --release --short | cut -d. -f1)u1
elif [ "$distributor" == "Ubuntu" ]; then
distname=~$(lsb_release --codename --short)1
fi
codename=$(lsb_release --codename --short)
date=$(date +%a," "%d" "%b" "%Y" "%T)
pkg_name=keydb-$majorv:$version$distname
# create build tree
cd ../../../
tar -czvf keydb_$version.orig.tar.gz --force-local KeyDB
cd KeyDB/pkg/deb/
mkdir -p $pkg_name/tmp
if [[ "$codename" == "xenial" ]] || [[ "$codename" == "stretch" ]]; then
cp -r debian_dh9 $pkg_name/tmp/debian
else
cp -r debian $pkg_name/tmp
fi
cp master_changelog $pkg_name/tmp/debian/changelog
mv ../../../keydb_$version.orig.tar.gz ./$pkg_name
cd $pkg_name/tmp
changelog_str="keydb ($majorv:$version-$build$distname) $codename; urgency=medium\n\n * $version $changelog_comments \n\n -- Ben Schermel <ben@eqalpha.com> $date +0000\n\n"
if [ $# -eq 0 ]; then
sed -i "1s/^/$changelog_str\n/" debian/changelog
elif [ $# -eq 1 ] && [ "$1" != "None" ]; then
sed -i "1s/^/$changelog_str\n/" debian/changelog
fi
sed -i "s/distribution_placeholder/$distname/g" debian/changelog
sed -i "s/codename_placeholder/$codename/g" debian/changelog
# generate required files and structure for pbuilder including .dsc file
debuild -S -sa
cd ../
# create pbuilder chrooted environment and build the deb package
if [ "$codename" == "xenial" ]; then
sudo pbuilder create --distribution $codename --othermirror "deb http://archive.ubuntu.com/ubuntu $codename universe multiverse"
else
sudo pbuilder create --distribution $codename
fi
sudo pbuilder --update
sudo pbuilder --build *.dsc
# move new packages to deb_files_generated and clean up
cp /var/cache/pbuilder/result/*$version*.deb ../deb_files_generated
sudo pbuilder clean
cd ../
rm -rf $pkg_name
Whitespace-only changes.
@@ -0,0 +1,30 @@
# -*- sh -*-
#
# Bash completion function for the 'keydb-cli' command.
#
# Steve
# --
# http://www.steve.org.uk
#
_keydb-cli()
{
COMPREPLY=()
cur=${COMP_WORDS[COMP_CWORD]}
prev=${COMP_WORDS[COMP_CWORD-1]}
#
# All known commands accepted. Sorted.
#
opts='bgrewriteaof bgsave dbsize debug decr decrby del echo exists expire expireat flushall flushdb get getset incr incrby info keys lastsave lindex llen lpop lpush lrange lrem lset ltrim mget move mset msetnx ping randomkey rename renamenx rewriteaof rpop rpoplpush rpush sadd save scard sdiff sdiffstore select set setnx shutdown sinter sinterstore sismember slaveof smembers smove sort spop srandmember srem sunion sunionstore ttl type zadd zcard zincrby zrange zrangebyscore zrem zremrangebyscore zrevrange zscore'
#
# Only complete on the first term.
#
if [ $COMP_CWORD -eq 1 ]; then
COMPREPLY=( $(compgen -W "${opts}" -- ${cur}) )
return 0
fi
}
complete -F _keydb-cli keydb-cli
+129
View File
@@ -0,0 +1,129 @@
#!/bin/sh
SYSTEMD_VER=$(systemctl --version | head -1 | cut -d' ' -f2)
if [ ${SYSTEMD_VER} -lt 237 ]
then
SYSTEMD_EXTRA=""
else
SYSTEMD_EXTRA=$(cat <<EOF
RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX
#MemoryDenyWriteExecute=true
ProtectKernelModules=true
ProtectKernelTunables=true
ProtectControlGroups=true
RestrictRealtime=true
RestrictNamespaces=true
EOF
)
fi
for BINARY in keydb-server keydb-sentinel
do
for MODE in default templated
do
case "${BINARY}" in
keydb-server)
NAME="keydb"
;;
keydb-sentinel)
NAME="sentinel"
;;
esac
case "${MODE}" in
default)
EXTRA="Alias=${NAME}.service"
TARGET="debian/${BINARY}.service"
NAMESPACED="${NAME}"
DESCRIPTION="Advanced key-value store"
;;
templated)
EXTRA=""
TARGET="debian/${BINARY}@.service"
NAMESPACED="${NAME}-%i"
DESCRIPTION="Advanced key-value store (%I)"
;;
esac
: >${TARGET}
if [ "${MODE}" = "templated" ]
then
cat >> ${TARGET} <<EOF
# Templated service file for ${BINARY}(1)
#
# Each instance of ${BINARY} requires its own configuration file:
#
# $ cp /etc/keydb/${NAME}.conf /etc/keydb/${NAME}-myname.conf
# $ chown keydb:keydb /etc/keydb/${NAME}-myname.conf
#
# Ensure each instance is using their own database:
#
# $ sed -i -e 's@^dbfilename .*@dbfilename dump-myname.rdb@' /etc/keydb/${NAME}-myname.conf
#
# We then listen exlusively on UNIX sockets to avoid TCP port collisions:
#
# $ sed -i -e 's@^port .*@port 0@' /etc/keydb/${NAME}-myname.conf
# $ sed -i -e 's@^\\(# \\)\\{0,1\\}unixsocket .*@unixsocket /var/run/${NAME}-myname/${BINARY}.sock@' /etc/keydb/${NAME}-myname.conf
#
# ... and ensure we are logging, etc. in a unique location:
#
# $ sed -i -e 's@^logfile .*@logfile /var/log/keydb/${BINARY}-myname.log@' /etc/keydb/${NAME}-myname.conf
# $ sed -i -e 's@^pidfile .*@pidfile /var/run/keydb-myname/${BINARY}.pid@' /etc/keydb/${NAME}-myname.conf
#
# We can then start the service as follows, validating we are using our own
# configuration:
#
# $ systemctl start ${BINARY}@myname.service
# $ keydb-cli -s /var/run/${NAME}-myname/${BINARY}.sock info | grep config_file
EOF
fi
cat >> ${TARGET} <<EOF
[Unit]
Description=${DESCRIPTION}
After=network.target
Documentation=https://docs.keydb.dev, man:${BINARY}(1)
[Service]
Type=forking
ExecStart=/usr/bin/${BINARY} /etc/keydb/${NAMESPACED}.conf
ExecStop=/bin/kill -s TERM \$MAINPID
PIDFile=/var/run/${NAMESPACED}/${BINARY}.pid
TimeoutStopSec=0
Restart=always
User=keydb
Group=keydb
RuntimeDirectory=${NAMESPACED}
RuntimeDirectoryMode=2755
UMask=007
PrivateTmp=yes
LimitNOFILE=65535
PrivateDevices=yes
ProtectHome=yes
ReadOnlyDirectories=/
ReadWriteDirectories=-/var/lib/keydb
ReadWriteDirectories=-/var/log/keydb
ReadWriteDirectories=-/var/run/${NAMESPACED}
NoNewPrivileges=true
CapabilityBoundingSet=CAP_SETGID CAP_SETUID CAP_SYS_RESOURCE${SYSTEMD_EXTRA}
# ${BINARY} can write to its own config file when in cluster mode so we
# permit writing there by default. If you are not using this feature, it is
# recommended that you replace the following lines with "ProtectSystem=full".
ProtectSystem=true
ReadWriteDirectories=-/etc/keydb
[Install]
WantedBy=multi-user.target
EOF
if [ "${EXTRA}" != "" ]
then
echo "${EXTRA}" >> "${TARGET}"
fi
done
done
+32
View File
@@ -0,0 +1,32 @@
keydb (5:5.3.3-1~bionic1) bionic; urgency=medium
* 5.3.3 Updating deb source package and naming convention.
-- Ben Schermel <ben@eqalpha.com> Fri, 25 Oct 2019 8:00:37 +0000
keydb (5:5.1.12-1chl1~bionic1) bionic; urgency=medium
* 5.1.1 update. This update fixes several rare deadlock scenarios. Deadlock detection is also added.
-- Ben Schermel <ben@eqalpha.com> Fri, 25 Oct 2019 8:00:37 +0000
keydb (5:5.1.11-1chl1~bionic1) bionic; urgency=medium
* 5.1 release. This release includes subkey expires (EXPIREMEMBER/EXPIREMEMBERAT), with updates to PTTL/TTL accordingly. New OBJECT LASTMODIFIED, BITIOP LSHIFT & BITOP RSHIFT commands. See https://docs.keydb.dev/blog/2019/10/20/blog-post/ for detailed review of release.
-- Ben Schermel <ben@eqalpha.com> Mon, 21 Oct 2019 8:00:37 +0000
keydb (5:5.0.1-1chl1~bionic1) bionic; urgency=medium
* Arm build now included for bionic package
-- Ben Schermel <ben@eqalpha.com> Wed, 21 Aug 2019 22:58:37 +0000
keydb (5:5.0.0-1chl1~bionic1) bionic; urgency=medium
* Initial release of KeyDB PPA. This PPA was originally derived from https://launchpad.net/~chris-lea/+archive/ubuntu/redis-server
-- Ben Schermel <ben@eqalpha.com> Wed, 21 Aug 2019 2:58:37 +0000
+1
View File
@@ -0,0 +1 @@
9
+77
View File
@@ -0,0 +1,77 @@
Source: keydb
Section: database
Priority: optional
Maintainer: Ben Schermel <ben@eqalpha.com>
Build-Depends:
debhelper (>= 9~),
dpkg-dev (>= 1.17.5),
systemd,
procps <!nocheck>,
build-essential <!nocheck>,
tcl <!nocheck>,
tcl-dev <!nocheck>,
uuid-dev <!nocheck>,
libcurl4-openssl-dev <!nocheck>,
nasm <!nocheck>,
autotools-dev <!nocheck>,
autoconf <!nocheck>,
libjemalloc-dev <!nocheck>,
libssl-dev <!nocheck>
Standards-Version: 4.2.1
Homepage: https://docs.keydb.dev/
Vcs-Git: https://github.com/JohnSully/KeyDB.git
Vcs-Browser: https://github.com/JohnSully/KeyDB
Package: keydb
Architecture: all
Depends:
keydb-server (<< ${binary:Version}.1~),
keydb-server (>= ${binary:Version}),
${misc:Depends},
Description: Persistent key-value database with network interface (metapackage)
keydb is a key-value database in a similar vein to memcache but the dataset
is non-volatile. keydb additionally provides native support for atomically
manipulating and querying data structures such as lists and sets.
.
The dataset is stored entirely in memory and periodically flushed to disk.
.
This package depends on the keydb-server package.
Package: keydb-sentinel
Architecture: any
Depends:
lsb-base (>= 3.2-14),
keydb-tools (= ${binary:Version}),
${misc:Depends},
Description: Persistent key-value database with network interface (monitoring)
keydb is a key-value database in a similar vein to memcache but the dataset
is non-volatile. keydb additionally provides native support for atomically
manipulating and querying data structures such as lists and sets.
.
This package contains the keydb Sentinel monitoring software.
Package: keydb-server
Architecture: any
Depends:
lsb-base (>= 3.2-14),
keydb-tools (= ${binary:Version}),
${misc:Depends},
Description: Persistent key-value database with network interface
keydb is a key-value database in a similar vein to memcache but the dataset
is non-volatile. keydb additionally provides native support for atomically
manipulating and querying data structures such as lists and sets.
.
The dataset is stored entirely in memory and periodically flushed to disk.
Package: keydb-tools
Architecture: any
Depends:
adduser,
${misc:Depends},
${shlibs:Depends},
Description: Persistent key-value database with network interface (client)
keydb is a key-value database in a similar vein to memcache but the dataset
is non-volatile. keydb additionally provides native support for atomically
manipulating and querying data structures such as lists and sets.
.
This package contains the command line client and other tools.
+61
View File
@@ -0,0 +1,61 @@
Format: https://www.debian.org/doc/packaging-manuals/copyright-format/1.0/
Upstream-Contact: John Sully <john@eqalpha.com>
Upstream-Name: keydb-pro
Source: https://github.com/JohnSully/KeyDB-Pro
Files: *
Copyright: © 2006-2014 Salvatore Sanfilippo <antirez@gmail.com>
Copyright © 2019, John Sully
License: Proprietary
Files: debian/*
Copyright: © 2009 Chris Lamb <lamby@debian.org>
License: BSD-3-clause
License: BSD-2-clause
Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are met:
.
1. Redistributions of source code must retain the above copyright notice, this
list of conditions and the following disclaimer.
.
2. Redistributions in binary form must reproduce the above copyright notice,
this list of conditions and the following disclaimer in the documentation
and/or other materials provided with the distribution.
.
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
License: BSD-3-clause
Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions
are met:
.
* Redistributions of source code must retain the above copyright
notice, this list of conditions and the following disclaimer.
* Redistributions in binary form must reproduce the above copyright
notice, this list of conditions and the following disclaimer in the
documentation and/or other materials provided with the distribution.
* Neither the name of Redis nor the names of its contributors may be
used to endorse or promote products derived from this software
without specific prior written permission.
.
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE
FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+68
View File
@@ -0,0 +1,68 @@
.TH KEYDB-BENCHMARK 1 "August 17, 2019"
.SH NAME
keydb-benchmark \- Benchmark a KeyDB instance
.SH SYNOPSIS
.B redis-benchmark
[\-h <host>] [\-p <port>] [\-c <clients>] [\-n <requests]> [\-k <boolean>]
.SH DESCRIPTION
KeyDB is a key-value database. It is similar to memcached but the dataset is
not volatile and other datatypes (such as lists and sets) are natively
supported.
.SH OPTIONS
.TP
\-h hostname
Server hostname (default 127.0.0.1)
.TP
\-p port
Server port (default 6379)
.TP
\-s socket
Server socket (overrides host and port)
.TP
\-a password
Password for KeyDB Auth
.TP
\-c clients
Number of parallel connections (default 50)
.TP
\-n requests
Total number of requests (default 100000)
.TP
\-d size
Data size of SET/GET value in bytes (default 2)
.TP
\-dbnum db
SELECT the specified db number (default 0)
.TP
\-k boolean
1=keep alive 0=reconnect (default 1)
.TP
\-r keyspacelen
Use random keys for SET/GET/INCR, random values for SADD Using this option the
benchmark will get/set keys in the form mykey_rand000000012456 instead of
constant keys, the <keyspacelen> argument determines the max number of values
for the random number. For instance if set to 10 only rand000000000000 -
rand000000000009 range will be allowed.
.TP
\-P numreq
Pipeline <numreq> requests. Default 1 (no pipeline).
.TP
\-q
Quiet. Just show query/sec values
.TP
\-\-csv
Ourput in CSV format
.TP
\-l
Loop. Run the tests forever
.TP
\-I
Idle mode. Just open N idle connections and wait.
.TP
\-D
Debug mode. more verbose.
.SH AUTHOR
\fBkeydb-benchmark\fP was written by John Sully, originating as a fork of Redis. Redis was written by Salvatore Sanfilippo.
.PP
This manual page was written by Chris Lamb <lamby@debian.org> for the Debian
project (but may be used by others). Modified by Ben Schermel <ben@eqalpha.com>
+17
View File
@@ -0,0 +1,17 @@
.TH KEYDB-CHECK-AOF 1 "August 17, 2019"
.SH NAME
keydb-check-aof \- Check integrity of a KeyDB .AOF file
.SH SYNOPSIS
.B keydb-check-aof
filename
.SH DESCRIPTION
KeyDB is a key-value database. It is similar to memcached but the dataset is
not volatile and other datatypes (such as lists and sets) are natively
supported.
.PP
This utility checks the integrity of a dumped .AOF file.
.SH AUTHOR
\fBkeydb-check-aof\fP was written by John Sully, originating as a fork of Redis. Redis was written by Salvatore Sanfilippo.
.PP
This manual page was written by Chris Lamb <lamby@debian.org> for the Debian
project (but may be used by others). Modified by Ben Schermel <ben@eqalpha.com>
+17
View File
@@ -0,0 +1,17 @@
.TH KEYDB-CHECK-RDB 1 "August 17, 2019"
.SH NAME
keydb-check-rdb \- Check integrity of KeyDB dumped database file
.SH SYNOPSIS
.B keydb-check-rdb
filename
.SH DESCRIPTION
KeyDB is a key-value database. It is similar to memcached but the dataset is
not volatile and other datatypes (such as lists and sets) are natively
supported.
.PP
This utility checks the integrity of a dumped database file.
.SH AUTHOR
\fBredis-check-rdb\fP was written by John Sully, originating as a fork of Redis. Redis was written by Salvatore Sanfilippo.
.PP
This manual page was written by Chris Lamb <lamby@debian.org> for the Debian
project (but may be used by others). Modified by Ben Schermel <ben@eqalpha.com>
+19
View File
@@ -0,0 +1,19 @@
.TH KEYDB-CLI 1 "August 17, 2019"
.SH NAME
keydb-cli \- Command-line client to keydb-server
.SH SYNOPSIS
.B keydb-cli
.RI [options]
.SH DESCRIPTION
KeyDB is a key-value database. It is similar to memcached but the dataset is
not volatile and other datatypes (such as lists and sets) are natively
supported.
.PP
\fBkeydb-cli\fP provides a simple command-line interface to a KeyDB server.
.SH OPTIONS
See \fBkeydb-doc\fP for more information on the commands KeyDB accepts.
.SH AUTHOR
\fBkeydb-cli\fP was written by John Sully, originating as a fork of Redis. Redis was written by Salvatore Sanfilippo.
.PP
This manual page was written by Chris Lamb <lamby@debian.org> for the Debian
project (but may be used by others). Modified by Ben Schermel <ben@eqalpha.com>
+23
View File
@@ -0,0 +1,23 @@
.TH KEYDB-SENTINEL 1 "August 17, 2019"
.SH NAME
keydb-sentinel \- Persistent key-value database (cluster mode)
.SH SYNOPSIS
.B keydb-sentinel
.RI configfile
.SH DESCRIPTION
KeyDB is a key-value database. It is similar to memcached but the dataset is
not volatile and other datatypes (such as lists and sets) are natively
supported.
.PP
.SH OPTIONS
.IP "configfile"
Read options from specified configuration file.
.SH NOTES
On Debian GNU/Linux systems, \fBkeydb-sentinel\fP is typically started via the
\fB/etc/init.d/keydb-sentinel\fP initscript, not manually. This defaults to using
\fB/etc/keydb/sentinel.conf\fP as a configuration file.
.SH AUTHOR
\fBkeydb-sentinel\fP was written by John Sully, originating as a fork of Redis. Redis was written by Salvatore Sanfilippo.
.PP
This manual page was written by Chris Lamb <lamby@debian.org> for the Debian
project (but may be used by others). Modified by Ben Schermel <ben@eqalpha.com>
+2
View File
@@ -0,0 +1,2 @@
debian/keydb-sentinel.service /lib/systemd/system/
pkg/deb/conf/sentinel.conf /etc/keydb
+7
View File
@@ -0,0 +1,7 @@
/var/log/keydb/keydb-sentinel*.log {
weekly
missingok
rotate 12
compress
notifempty
}
+1
View File
@@ -0,0 +1 @@
debian/keydb-sentinel.1
+26
View File
@@ -0,0 +1,26 @@
#!/bin/sh
set -eu
USER="keydb"
GROUP="$USER"
CONFFILE="/etc/keydb/sentinel.conf"
if [ "$1" = "configure" ]
then
if ! dpkg-statoverride --list ${CONFFILE} >/dev/null 2>&1
then
dpkg-statoverride --update --add ${USER} ${GROUP} 640 ${CONFFILE}
fi
fi
#DEBHELPER#
if [ "$1" = "configure" ]
then
find /etc/keydb -maxdepth 1 -type d -name 'keydb-sentinel.*.d' -empty -delete
fi
systemctl daemon-reload
exit 0
+16
View File
@@ -0,0 +1,16 @@
#!/bin/sh
set -eu
CONFFILE="/etc/keydb/sentinel.conf"
if [ "$1" = "purge" ]
then
dpkg-statoverride --remove ${CONFFILE} || test $? -eq 2
fi
#DEBHELPER#
systemctl daemon-reload
exit 0
+23
View File
@@ -0,0 +1,23 @@
.TH KEYDB-SERVER 1 "August 17, 2019"
.SH NAME
keydb-server \- Persistent key-value database
.SH SYNOPSIS
.B keydb-server
.RI configfile
.SH DESCRIPTION
KeyDB is a key-value database. It is similar to memcached but the dataset is
not volatile and other datatypes (such as lists and sets) are natively
supported.
.PP
.SH OPTIONS
.IP "configfile"
Read options from specified configuration file.
.SH NOTES
On Debian GNU/Linux systems, \fBkeydb-server\fP is typically started via the
\fB/etc/init.d/keydb-server\fP initscript, not manually. This defaults to using
\fB/etc/keydb/keydb.conf\fP as a configuration file.
.SH AUTHOR
\fBkeydb-server\fP was written by John Sully, originating as a fork of Redis. Redis was written by Salvatore Sanfilippo.
.PP
This manual page was written by Chris Lamb <lamby@debian.org> for the Debian
project (but may be used by others). Modified by Ben Schermel <ben@eqalpha.com>
+1
View File
@@ -0,0 +1 @@
README.md
+2
View File
@@ -0,0 +1,2 @@
debian/keydb-server.service /lib/systemd/system/
pkg/deb/conf/keydb.conf /etc/keydb
+7
View File
@@ -0,0 +1,7 @@
/var/log/keydb/keydb-server*.log {
weekly
missingok
rotate 12
compress
notifempty
}
+1
View File
@@ -0,0 +1 @@
debian/keydb-server.1
+26
View File
@@ -0,0 +1,26 @@
#!/bin/sh
set -eu
USER="keydb"
GROUP="$USER"
CONFFILE="/etc/keydb/keydb.conf"
if [ "$1" = "configure" ]
then
if ! dpkg-statoverride --list ${CONFFILE} >/dev/null 2>&1
then
dpkg-statoverride --update --add ${USER} ${GROUP} 640 ${CONFFILE}
fi
fi
#DEBHELPER#
if [ "$1" = "configure" ]
then
find /etc/keydb -maxdepth 1 -type d -name 'keydb-server.*.d' -empty -delete
fi
systemctl daemon-reload
exit 0
+16
View File
@@ -0,0 +1,16 @@
#!/bin/sh
set -eu
CONFFILE="/etc/keydb/keydb.conf"
if [ "${1}" = "purge" ]
then
dpkg-statoverride --remove ${CONFFILE} || test $? -eq 2
fi
#DEBHELPER#
systemctl daemon-reload
exit 0
+2
View File
@@ -0,0 +1,2 @@
src/redis-trib.rb
utils/lru
+7
View File
@@ -0,0 +1,7 @@
debian/bash_completion.d/* /usr/share/bash-completion/completions
src/keydb-benchmark /usr/bin
src/keydb-check-aof /usr/bin
src/keydb-check-rdb /usr/bin
src/keydb-cli /usr/bin
src/keydb-server /usr/bin
src/keydb-sentinel /usr/bin
+4
View File
@@ -0,0 +1,4 @@
debian/keydb-benchmark.1
debian/keydb-check-aof.1
debian/keydb-check-rdb.1
debian/keydb-cli.1
+47
View File
@@ -0,0 +1,47 @@
#!/bin/sh
set -eu
USER="keydb"
Setup_dir () {
DIR="${1}"
MODE="${2}"
GROUP="${3}"
mkdir -p ${DIR}
case "${DIR}" in
/var/log/keydb)
MODE="02750"
GROUP="adm"
;;
*)
MODE="750"
GROUP="${USER}"
;;
esac
if ! dpkg-statoverride --list ${DIR} >/dev/null 2>&1
then
chown ${USER}:${GROUP} ${DIR}
chmod ${MODE} ${DIR}
fi
}
if [ "$1" = "configure" ]
then
adduser \
--system \
--home /var/lib/keydb \
--quiet \
--group \
${USER} || true
Setup_dir /var/log/keydb ${USER}:adm 2750
Setup_dir /var/lib/keydb ${USER}:${USER} 750
fi
#DEBHELPER#
exit 0
+13
View File
@@ -0,0 +1,13 @@
#!/bin/sh
set -eu
if [ "${1}" = "purge" ]
then
userdel keydb || true
rm -rf /var/lib/keydb /var/log/keydb
fi
#DEBHELPER#
exit 0
+39
View File
@@ -0,0 +1,39 @@
#!/usr/bin/make -f
include /usr/share/dpkg/buildflags.mk
export BUILD_TLS=yes
export CFLAGS CPPFLAGS LDFLAGS
export DEB_BUILD_MAINT_OPTIONS = hardening=+all
export DEB_LDFLAGS_MAINT_APPEND = -ldl -latomic $(LUA_LDFLAGS)
ifneq (,$(filter parallel=%,$(DEB_BUILD_OPTIONS)))
NUMJOBS = $(patsubst parallel=%,%,$(filter parallel=%,$(DEB_BUILD_OPTIONS)))
MAKEFLAGS += -j$(NUMJOBS)
export MAKEFLAGS
endif
%:
dh $@
override_dh_auto_install:
debian/bin/generate-systemd-service-files
dh_installsystemd --restart-after-upgrade
override_dh_auto_test:
ifeq (,$(filter nocheck,$(DEB_BUILD_OPTIONS)))
# Avoid race conditions in upstream testsuite.
./runtest || true
./runtest-cluster || true
./runtest-sentinel || true
endif
override_dh_auto_clean:
dh_auto_clean
rm -f src/release.h debian/*.service
override_dh_compress:
dh_compress -Xredis-trib.rb
override_dh_installchangelogs:
dh_installchangelogs --keep 00-RELEASENOTES
+1
View File
@@ -0,0 +1 @@
3.0 (quilt)
+2
View File
@@ -0,0 +1,2 @@
# Upstream do not provide signed tarballs.
keydb-pro source: debian-watch-does-not-check-gpg-signature
+8
View File
@@ -0,0 +1,8 @@
#!/bin/sh
#
# Show the INFO from "keydb-cli"
set -eu
keydb-cli INFO
keydb-cli LOLWUT
+7
View File
@@ -0,0 +1,7 @@
#!/bin/sh
#
# Run the benchmarking
set -eu
keydb-benchmark -P 10
+5
View File
@@ -0,0 +1,5 @@
#!/bin/sh
#
# Smoke test keydb-check-aof
keydb-check-aof 2>&1 | grep -qsi usage:
+10
View File
@@ -0,0 +1,10 @@
#!/bin/sh
#
# Test keydb-check-rdb
set -eu
# Perform a synchronous save to ensure .rdb file eixsts
keydb-cli SAVE
keydb-check-rdb /var/lib/keydb/dump.rdb
+8
View File
@@ -0,0 +1,8 @@
Tests: 0001-keydb-cli
Tests: 0002-benchmark
Tests: 0003-keydb-check-aof
Tests: 0004-keydb-check-rdb
Restrictions: needs-root
Loaded 100 of 406 files, more files were not shown because too many files have changed in this diff. Show more