GitLab

* Adds StartOffset to ReaderObjectAttrs, which for range
requests is the value from which the read began
* Updates the range reader integration tests to also test
negative offsets aka offsets from the end of the file
* Adds integration tests for the various styles of reading
the same last n bytes from a file
* Adds example tests to show the various styles of using
(*ObjectHandle).NewRangeReader
* Replaces CL https://code-review.googlesource.com/c/gocloud/+/44132
for which the original author @brimworks no longer has
Gerrit access due to job changes

Updates #1540.

Change-Id: I56b7777cc46e3c501593e92e27a7c083a729fed5
Reviewed-on: https://code-review.googlesource.com/c/gocloud/+/44610

Reviewed-by: Frank Natividad <[email protected]>
Reviewed-by: kokoro <[email protected]>

Change-Id: Ic50f1db5f36c0b3d81aba75f3b792151e35ccf31
Reviewed-on: https://code-review.googlesource.com/c/gocloud/+/44670

Reviewed-by: kokoro <[email protected]>
Reviewed-by: Jean de Klerk <[email protected]>

Fixes: #1550

Change-Id: I47917416f8e26f0d95d4828feadf95135bb2b056
Reviewed-on: https://code-review.googlesource.com/c/gocloud/+/44550

Reviewed-by: kokoro <[email protected]>
Reviewed-by: Tyler Bui-Palsulich <[email protected]>

Changes to address bug 117902632.

Change-Id: I0cc38a6103b32f36cc7e93cb4e7db857427b5dd0
Reviewed-on: https://code-review.googlesource.com/c/gocloud/+/44590

Reviewed-by: kokoro <[email protected]>
Reviewed-by: Igor Bernstein <[email protected]>

Change-Id: Ifc564eb5a5953bf5740ecd9e8ec0b3a431d99cc5
Reviewed-on: https://code-review.googlesource.com/c/gocloud/+/44570

Reviewed-by: kokoro <[email protected]>
Reviewed-by: Emmanuel Odeke <[email protected]>

The RPC's in replenishPool used the context with timeout that was
passed in by the maintainer method. By default, this would have a
timeout of 1 minute, but can be configured to a different value.
This timeout value should not be used to timeout the gRPC calls,
as a very low value would start timing out all or most gRPC calls
for creating and deleting sessions in replenishPool, causing the
method to report a lot of timeout error and preventing it from
executing its actual task. The TestStressSessionPool tests the
session pool with different configurations. One of them is with a
healthCheckSampleInterval of 10ms, which currently fills up the
test log with a large number of timeout errors (when executed
with the -race argument).

Change-Id: I47bb58185f2f9768dc96aa50e2c420cbe31bac7a
Reviewed-on: https://code-review.googlesource.com/c/gocloud/+/44177

Reviewed-by: kokoro <[email protected]>
Reviewed-by: Jean de Klerk <[email protected]>

TESTED:

$ GCLOUD_TESTS_GOLANG_PROJECT_ID=dulcet-port-762 GCLOUD_TESTS_GOLANG_PROFILER_ZONE=us-west1-b go test -timeout=60m -v ./...

Change-Id: I8ea0c7f264f634356a0fe8c1e00360cca8fd71fb
Reviewed-on: https://code-review.googlesource.com/c/gocloud/+/44430

Reviewed-by: kokoro <[email protected]>
Reviewed-by: Kalyana Chadalavada <[email protected]>

Google Cloud Storage supports reading the last N bytes from a file
if the Range header is given a negative offset like:
    Range: bytes=-N
where N is an integer, as per
 https://cloud.google.com/storage/docs/xml-api/reference-headers#range

This CL adds that support to NewRangeReader and requires users
to pass in a negative offset and a negative length for example:

    r, err := obj.NewRangeReader(ctx, -10, -1)

Fixes #1540

Change-Id: Ia6803713334516053994b5a7f1af9b8d5d9cd586
Reviewed-on: https://code-review.googlesource.com/c/gocloud/+/44131

Reviewed-by: kokoro <[email protected]>
Reviewed-by: Frank Natividad <[email protected]>
Reviewed-by: Emmanuel Odeke <[email protected]>

Removed 'an an' and 'the the', 'the a' in comment.

Change-Id: I01a1f12e44bc8a2db5246e92fd83da2d7c1749e7
Reviewed-on: https://code-review.googlesource.com/c/gocloud/+/44470

Reviewed-by: kokoro <[email protected]>
Reviewed-by: Emmanuel Odeke <[email protected]>

The readily mocked inmem Spanner server was included in the normal
build, instead of only being included in test builds. The test server
has been moved to the internal testutil package, the package has been
renamed as testutil_test, and the setup for a test client has been
added to the client_test.go file to prevent a circular import.

Fixes #1539

Change-Id: I56dc57345ef65dd2bebd57742961848e7abe1818
Reviewed-on: https://code-review.googlesource.com/c/gocloud/+/44176

Reviewed-by: kokoro <[email protected]>
Reviewed-by: Jean de Klerk <[email protected]>

Changes to address docs bugs 120855255, 122635510, 31671073, 74213421,

112533440, 119277645, and 119277663.

Change-Id: I55834776c0b4c7274c45e2e653c2473549a643d9
Reviewed-on: https://code-review.googlesource.com/c/gocloud/+/44370

Reviewed-by: kokoro <[email protected]>
Reviewed-by: Tyler Bui-Palsulich <[email protected]>

This is faster for non-trivial tables for many operations (because we
can use binary searches), and is a better match for production Spanner.

Change-Id: I2a3271a8bca1e466a923a1367d2220615a6e1328
Reviewed-on: https://code-review.googlesource.com/c/gocloud/+/44351

Reviewed-by: kokoro <[email protected]>
Reviewed-by: Knut Olav Løite <[email protected]>

Change-Id: I4caf85e1a090fce3a50076384b0e90ec92a52d64
Reviewed-on: https://code-review.googlesource.com/c/gocloud/+/44350

Reviewed-by: kokoro <[email protected]>
Reviewed-by: Knut Olav Løite <[email protected]>

The value -9223372036854775808 was not parsed correctly as it
was first parsed as a positive Int64 and later flipped to a
negative number.

Fixes #1541

Change-Id: I627c2c4fa90a5b3b8f4fe56d8ba8ec96a7018f80
Reviewed-on: https://code-review.googlesource.com/c/gocloud/+/44178

Reviewed-by: kokoro <[email protected]>
Reviewed-by: David Symonds <[email protected]>

Change-Id: I434182a0c7de081beb81a538e12a154fd0b5bec9
Reviewed-on: https://code-review.googlesource.com/c/gocloud/+/44290

Reviewed-by: Emmanuel Odeke <[email protected]>

We can reuse compareVals, which also handles NULLs correctly.

Change-Id: Ia69d840103a508b6c5d9d90da989bc3ba7f7721b
Reviewed-on: https://code-review.googlesource.com/c/gocloud/+/44238

Reviewed-by: kokoro <[email protected]>
Reviewed-by: Knut Olav Løite <[email protected]>

Change-Id: I228b5d90979db14d3d51a08ee940f9e077834d69
Reviewed-on: https://code-review.googlesource.com/c/gocloud/+/44237

Reviewed-by: kokoro <[email protected]>
Reviewed-by: Knut Olav Løite <[email protected]>

Also write some thorough tests for key range inclusion checking,
based on the examples given in google/spanner/v1/keys.proto.

Change-Id: I77fd5f7aecb607bfe7b7c22addf2d403867c928c
Reviewed-on: https://code-review.googlesource.com/c/gocloud/+/44236

Reviewed-by: kokoro <[email protected]>
Reviewed-by: Knut Olav Løite <[email protected]>

Change-Id: I36cbdd88a0a4f356f480cb3ba54e9b65c6a25468
Reviewed-on: https://code-review.googlesource.com/c/gocloud/+/44235

Reviewed-by: kokoro <[email protected]>
Reviewed-by: Knut Olav Løite <[email protected]>

There's no need to pass pkIndexes through to the closures, since the
passed row is already in table order, so the primary key is the initial
t.pkCols columns of that row.

Change-Id: I52c97ec5a4062a6d9d5878c46e28d65121cc1059
Reviewed-on: https://code-review.googlesource.com/c/gocloud/+/44234

Reviewed-by: kokoro <[email protected]>
Reviewed-by: Knut Olav Løite <[email protected]>

Instead of allocating a new slice to save monotonically
increasing sums on every invocation of (*D).Percentile,
on every invocation of New(n) to create a new D, we
allocate a []uint64 slice that will be used to store the
monotonically increasing sums thus avoiding that recomputation
every single time.

This change removes all allocations in (*D).Percentile.

name            old time/op    new time/op    delta
Distribution-8    1.68µs ± 3%    0.95µs ± 3%   -43.39%  (p=0.008 n=5+5)

name            old alloc/op   new alloc/op   delta
Distribution-8    8.19kB ± 0%    0.00kB       -100.00%  (p=0.008 n=5+5)

name            old allocs/op  new allocs/op  delta
Distribution-8      1.00 ± 0%      0.00       -100.00%  (p=0.008 n=5+5)

Fixes #1538

Change-Id: I421cf856bd16263b36117c2a7eece3df8c832773
Reviewed-on: https://code-review.googlesource.com/c/gocloud/+/44291

Reviewed-by: kokoro <[email protected]>
Reviewed-by: Emmanuel Odeke <[email protected]>

v0.9.0 features the fixes to properly use
resumable uploads as per
https://code-review.googlesource.com/c/google-api-go-client/+/44070

Change-Id: Icb07f2eb8a8689047ee9982561f6469a8deee7a7
Reviewed-on: https://code-review.googlesource.com/c/gocloud/+/44310

Reviewed-by: kokoro <[email protected]>
Reviewed-by: Alex Hong <[email protected]>

Change-Id: I0255cd68b67e8b7d26a8535d6ca000c362bbf665
Reviewed-on: https://code-review.googlesource.com/c/gocloud/+/44271

Reviewed-by: kokoro <[email protected]>
Reviewed-by: Emmanuel Odeke <[email protected]>

use subscription name to calculate msg size
remove unused constant in service.go

Fixes #1441

Change-Id: I03dd06e8ddb5ce76b9adbc306026d9d563587357
Reviewed-on: https://code-review.googlesource.com/c/gocloud/+/41710

Reviewed-by: kokoro <[email protected]>
Reviewed-by: Alex Hong <[email protected]>

Fixes #1537

Change-Id: I3259edb27e720f1067151307b4b192cbca7f2008
Reviewed-on: https://code-review.googlesource.com/c/gocloud/+/44270

Reviewed-by: kokoro <[email protected]>
Reviewed-by: Tyler Bui-Palsulich <[email protected]>

This doesn't implement any transactional behaviour, but merely allows
the read operation to proceed.

Change-Id: I73acba942ad6afef31e090cf092ebebf0c5cebf9
Reviewed-on: https://code-review.googlesource.com/c/gocloud/+/44229

Reviewed-by: kokoro <[email protected]>
Reviewed-by: Knut Olav Løite <[email protected]>

When we converted "files" to be
    map[string]io.Reader
we used an io.LimitedReader that then unfortunately
meant that multiple runs wouldn't stream back the
content. That's because when io.LimitedReader hits
io.EOF, we need to reset its N for it to read
more from rand.Reader.

This would then mean that the second request would
stream back 0 bytes, which is not what we want.

An update to make this reset would be a bit complex,
so instead make "files"
    map[string]int64
mapping the file name to its size. When a request
is then made for the file, we'll create a fresh
io.LimitedReader, which produces the correct behavior.

Change-Id: I609da9458fc28efece4967a73520b5ac6214f6cb
Reviewed-on: https://code-review.googlesource.com/c/gocloud/+/44191

Reviewed-by: kokoro <[email protected]>
Reviewed-by: Alex Hong <[email protected]>

Change-Id: I18cb5b18b232c8c1ec2e6396ce8c055ae1a114ec
Reviewed-on: https://code-review.googlesource.com/c/gocloud/+/44134

Reviewed-by: kokoro <[email protected]>
Reviewed-by: Jean de Klerk <[email protected]>

Buffers output instead of trying to send all at once, leading to less
broken pipes and the like.

Also change log.Exitf to log.Print so that a single bad request/response
interaction doesn't kill the entire server.

Change-Id: I278cda454e5cc1cf0de40385a6bfee79c2227c91
Reviewed-on: https://code-review.googlesource.com/c/gocloud/+/44133

Reviewed-by: kokoro <[email protected]>
Reviewed-by: Emmanuel Odeke <[email protected]>

The mapping to the correct Spanner type code was overlooked.

Change-Id: I9118caf02e43fcdaa98ca42f90deb0c931fcc41e
Reviewed-on: https://code-review.googlesource.com/c/gocloud/+/44110

Reviewed-by: kokoro <[email protected]>
Reviewed-by: Knut Olav Løite <[email protected]>

- Allow URLs for Go, Node, Java.
- Add 1mb and 100mb files.
- Add --port flag.
- Add README.md.

Change-Id: If650bf88e705de002cac31a82393c7da104b355e
Reviewed-on: https://code-review.googlesource.com/c/gocloud/+/44013

Reviewed-by: Emmanuel Odeke <[email protected]>
Reviewed-by: kokoro <[email protected]>

This package can auto-detect monitored resources on GCE.  This change
adds detection of monitored resources if running on GAE Standard (second
generation) too.

I got the label-names we need from [1] and the envvar names from [2]
(which is written for Go 1.12, but I believe applies to all second-gen
runtimes).

I wasn't sure if there was a useful way to write automated tests, but I
put similar code in my app (via the `CommonResource` option), and
checked that it caused my log lines to show up in the logs viewer under
"GAE Application" (and/or the relevant module/version) rather than
"Google Project".

[1] https://cloud.google.com/monitoring/api/resources#tag_gae_app
[2] https://cloud.google.com/appengine/docs/standard/go112/runtime#environment_variables

Fixes #1521.

Change-Id: Ie7243c16fb5b10ac06a57bdcb526c247b0e215be
Reviewed-on: https://code-review.googlesource.com/c/gocloud/+/42810

Reviewed-by: Steven Buss <[email protected]>
Reviewed-by: kokoro <[email protected]>
Reviewed-by: Emmanuel Odeke <[email protected]>

Fixes #1535

Change-Id: Icb44598adf4405943c3ec038997cbb37e024116d
Reviewed-on: https://code-review.googlesource.com/c/gocloud/+/44130

Reviewed-by: Tyler Bui-Palsulich <[email protected]>

Change-Id: Icaf5993fd0063d63d55bbb0eacd00119cfeb1901
Reviewed-on: https://code-review.googlesource.com/c/gocloud/+/44091

Reviewed-by: kokoro <[email protected]>
Reviewed-by: Jean de Klerk <[email protected]>

Added HMACKey listing ability on (*storage).Client.
By default, even deleted keys will be listed, so
a user should filter out such keys on their own.
Also serviceAccount filtration will be added later on.

Updates #1486

Change-Id: Ida88fe483c2ecf291d57b41bd20a90dbab5784e3
Reviewed-on: https://code-review.googlesource.com/c/gocloud/+/43650

Reviewed-by: kokoro <[email protected]>
Reviewed-by: Jean de Klerk <[email protected]>

Change-Id: Ibdae2425ce3a0c9c323531b434cfe8aa706654e9
Reviewed-on: https://code-review.googlesource.com/c/gocloud/+/44090

Reviewed-by: Emmanuel Odeke <[email protected]>

- Rename certificate/ to cert/
- Update the run location from `storage/internal/` to
`storage/internal/benchwrapper` by updating the relative path to certs
- Add --port option, per other benchwrappers (java, node)
- Increase read size from 4bytes to 1024*1024bytes
- Hard require STORAGE_EMULATOR_HOST.
- Add README.md with run instructions

Change-Id: Icaf96ed6b74ff83f4afe61e1efaa9efc4b0e01b2
Reviewed-on: https://code-review.googlesource.com/c/gocloud/+/44014

Reviewed-by: kokoro <[email protected]>
Reviewed-by: Tyler Bui-Palsulich <[email protected]>

Also updates all debug-agent licenses to LLC.

Change-Id: I0af63dd674ffeeb07c6eb37ca7608fabcdc16d5a
Reviewed-on: https://code-review.googlesource.com/c/gocloud/+/44012

Reviewed-by: kokoro <[email protected]>
Reviewed-by: Tyler Bui-Palsulich <[email protected]>
Reviewed-by: Emmanuel Odeke <[email protected]>

Run `go mod tidy` in repo root, logging/, datastore/, bigtable/, and bigquery/.

Also adds a basic script to tidy all modules.

Also updates RELEASING.md with releasing instructions.

Change-Id: I50b62691eb8e9a04f1225fdc460cee9063e919a0
Reviewed-on: https://code-review.googlesource.com/c/gocloud/+/44011

Reviewed-by: Tyler Bui-Palsulich <[email protected]>
Reviewed-by: kokoro <[email protected]>
Reviewed-by: Emmanuel Odeke <[email protected]>

Reasons for carve out:

- It allows us to communicate client stability with semver, instead of all
  clients being contained in a mega v0.Y.Z module.
- It allows us to separate the per-client dependencies.

This CL was created by:

1. Create go_mod_tidy_hack.go
2. Run:

    cd datastore
    go mod init cloud.google.com/go/bigtable
    go mod edit -require cloud.google.com/[email protected]
    go mod edit -replace cloud.google.com/[email protected]=../
    go mod tidy
    go mod edit -dropreplace cloud.google.com/[email protected]
    cd ..
    go mod tidy

This CL will be tagged v0.44.3 and bigtable/v1.0.0.

Change-Id: I79a70555591c42b243d397b64575220c7bfca240
Reviewed-on: https://code-review.googlesource.com/c/gocloud/+/43891

Reviewed-by: Tyler Bui-Palsulich <[email protected]>