目录
Krinkin, Mike

Use def_ref_codec_lib instead of including def_ref_code.{cc|hpp} as sources (#37)

def_ref_codec.cc is listed as a source of both object_codec_lib and def_ref_codec_lib targets. It does not cause problems for binaries that only depend on one of those targets, but when you start depending on both you might hit some issues.

Specifically, in Envoy, we have an extension that depends on //hessian2:codec_impl_lib and //hessian2/basic_codec:object_codec_lib (see https://github.com/envoyproxy/envoy/blob/d82453231f89cb021eb716a23805073cff1f7179/source/extensions/filters/network/dubbo_proxy/BUILD#L12-L22). As a result def_ref_codec.cc file is referenced as a source twice:

  1. as one of the sources of //hessian2/basic_codec:object_codec_lib
  2. as one of the sources of //hessian2/basic_codec:def_ref_codec_lib that is a dependency of //hesian2:codec_impl_lib.

Using the same source file multiple times as a source in Bazel does not seem to be common, but it also may cause issues.

For once, in C++ in principle having the same object file listed multiple times when linking may result in multiple definitions error (depending on the actual content of the object file), but in my case I hit a different issue.

I’m working on a project in which I have to build Envoy, but, unfortunately, I cannot use the Envoy CI image with all the tools preinstalled, as a result environment in which I build Envoy is somewhat different from the one provided by Envoy CI Docker image.

And when I try to build Envoy, at some point Bazel calls dwp tool to collect debugging information from dwo files into a single dwp file. The command Bazel uses to call dwp references def_ref_codec.dwo twice because def_ref_codec.cc is referenced twice as a source. dwp tool is not equipped to deal with that and complains about duplicate debug info and ultimately fails build:

error: duplicate DWO ID (852C89B14FE85A8F) in 'external/com_github_alibaba_hessian2_codec/hessian2/basic_codec/def_ref_codec.cc' (from 'bazel-out/k8-opt/bin/external/com_github_alibaba_hessian2_codec/hessian2/basic_codec/_objs/def_ref_codec_lib/def_ref_codec.dwo') and 'external/com_github_alibaba_hessian2_codec/hessian2/basic_codec/def_ref_codec.cc' (from 'bazel-out/k8-opt/bin/external/com_github_alibaba_hessian2_codec/hessian2/basic_codec/_objs/object_codec_lib/def_ref_codec.dwo')

I know that when you build Envoy with Envoy CI docker image the same issue does not happen, but after debugging for a few days, I still don’t quite understand why. I tried different versions of the dwp tool, but they all return the same error, which makes me think that the issue is likely not in the dwp tool itself.

Even though I still don’t know why it works currently in Envoy, it seems that this change makes sense (we should not list the same cc file multiple times as a source) and it should be safe (the same code is still listed as a dependency), so I figured that folks might be open to accept the change even with some unknowns.

Signed-off-by: Mikhail Krinkin krinkin.m.u@gmail.com

1年前28次提交

hessian2-codec

CI License Coverage

hessian2-codec is a C++ library from Alibaba for hessian2 codec. It is a complete C++ implementation of hessian2 spec. Because it was originally intended to implement the Dubbo Filter of Envoy, it did not provide good support for serialization of user-defined types (there is only one way to implement user-defined types using ADL, but it is not very complete and does not support nested types well). At the moment it is simply deserializing content into some C++ intermediate types.

Getting Started

Install

  1. To download and install Bazel (and any of its dependencies), consult the Bazel Installation Guide.
  2. Refer to Supported Platforms installation related compiler.
  3. Use hessian2-codec, see the demo directory for details.
$ cd demo
$ bazel build //...
$ ./bazel-bin/demo

Basic usage

#include <iostream>

#include "hessian2/codec.hpp"
#include "hessian2/basic_codec/object_codec.hpp"

int main() {
  {
    std::string out;
    ::Hessian2::Encoder encode(out);
    encode.encode<std::string>("test string");
    ::Hessian2::Decoder decode(out);
    auto ret = decode.decode<std::string>();
    if (ret) {
      std::cout << *ret << std::endl;
    } else {
      std::cerr << "decode failed: " << decode.getErrorMessage() << std::endl;
    }
  }
  {
    std::string out;
    ::Hessian2::Encoder encode(out);
    encode.encode<int64_t>(100);
    ::Hessian2::Decoder decode(out);
    auto ret = decode.decode<int64_t>();
    if (ret) {
      std::cout << *ret << std::endl;
    } else {
      std::cerr << "decode failed: " << decode.getErrorMessage() << std::endl;
    }
  }

  return 0;
}

Advance usage

  1. Implement the serialization and deserialization of custom types with ADL
#include <iostream>

#include "hessian2/codec.hpp"
#include "hessian2/basic_codec/object_codec.hpp"

struct Person {
  int32_t age_{0};
  std::string name_;
};

// The custom struct needs to implement from_hessian and to_hessian methods to
// encode and decode

void fromHessian(Person&, ::Hessian2::Decoder&);
bool toHessian(const Person&, ::Hessian2::Encoder&);

void fromHessian(Person& p, ::Hessian2::Decoder& d) {
  auto age = d.decode<int32_t>();
  if (age) {
    p.age_ = *age;
  }

  auto name = d.decode<std::string>();
  if (name) {
    p.name_ = *name;
  }
}

bool toHessian(const Person& p, ::Hessian2::Encoder& e) {
  e.encode<int32_t>(p.age_);
  e.encode<std::string>(p.name_);
  return true;
}

int main() {
  std::string out;
  Hessian2::Encoder encode(out);
  Person s;
  s.age_ = 12;
  s.name_ = "test";

  encode.encode<Person>(s);
  Hessian2::Decoder decode(out);
  auto decode_person = decode.decode<Person>();
  if (!decode_person) {
    std::cerr << "hessian decode failed " << decode.getErrorMessage()
              << std::endl;
    return -1;
  }
  std::cout << "Age: " << decode_person->age_
            << " Name: " << decode_person->name_ << std::endl;
}

There is currently no way to serialize container nested custom types such asstd::list<Person>.

  1. Customize Reader and Writer

Hessian2-codec uses the std::string implementation of reader and Writer by default, although we can customize both implementations.


#include <vector>
#include <iostream>

#include "hessian2/codec.hpp"
#include "hessian2/basic_codec/object_codec.hpp"
#include "hessian2/reader.hpp"
#include "hessian2/writer.hpp"

#include "absl/strings/string_view.h"

struct Slice {
  const uint8_t* data_;
  size_t size_;
};

class SliceReader : public ::Hessian2::Reader {
 public:
  SliceReader(Slice buffer) : buffer_(buffer){};
  virtual ~SliceReader() = default;

  virtual void rawReadNBytes(void* out, size_t len,
                             size_t peek_offset) override {
    ABSL_ASSERT(byteAvailable() + peek_offset >= len);
    uint8_t* dest = static_cast<uint8_t*>(out);
    // offset() Returns the current position that has been read.
    memcpy(dest, buffer_.data_ + offset() + peek_offset, len);
  }
  virtual uint64_t length() const override { return buffer_.size_; }

 private:
  Slice buffer_;
};

class VectorWriter : public ::Hessian2::Writer {
 public:
  VectorWriter(std::vector<uint8_t>& data) : data_(data) {}
  ~VectorWriter() = default;
  virtual void rawWrite(const void* data, uint64_t size) {
    const char* src = static_cast<const char*>(data);
    for (size_t i = 0; i < size; i++) {
      data_.push_back(src[i]);
    }
  }
  virtual void rawWrite(absl::string_view data) {
    for (auto& ch : data) {
      data_.push_back(ch);
    }
  }

 private:
  std::vector<uint8_t>& data_;
};

int main() {
  std::vector<uint8_t> data;
  auto writer = std::make_unique<VectorWriter>(data);

  ::Hessian2::Encoder encode(std::move(writer));
  encode.encode<std::string>("test string");
  Slice s{static_cast<const uint8_t*>(data.data()), data.size()};
  auto reader = std::make_unique<SliceReader>(s);
  ::Hessian2::Decoder decode(std::move(reader));
  auto ret = decode.decode<std::string>();
  if (ret) {
    std::cout << *ret << std::endl;
  } else {
    std::cerr << "decode failed: " << decode.getErrorMessage() << std::endl;
  }
}

Type mapping

C++ does not have a global parent like Java Object, so there is no single type that can represent all hessian types, so we create an Object base class from which all hessian types are inherited.

hessian type java type C++ type
null null NullObject
binary byte[] BinaryObject
boolean boolean BooleanObject
date java.util.Date DateObject
double double DoubleObject
int int IntegerObject
long long LongObject
string java.lang.String StringObject
untyped list java.util.List UntypedListObject
typed list java.util.ArrayList TypedListObject
untyped map java.util.Map UntypedMapObject
typed map for some OO language TypedMapObject
object custom define object ClassInstance

Supported Platforms

hessian2-codec requires a codebase and compiler compliant with the C++14 standard or newer.

The hessian2-codec code is officially supported on the following platforms. Operating systems or tools not listed below are community-supported. For community-supported platforms, patches that do not complicate the code may be considered.

If you notice any problems on your platform, please file an issue on the hessian2-codec GitHub Issue Tracker. Pull requests containing fixes are welcome!

Operating Systems

  • Linux
  • macOS
  • Windows(Theoretically, yes, but I haven’t tested it.)

Compilers

  • gcc 7.0+
  • clang 7.0+

Build Systems

Who Is Using hessian2-codec?

In addition to many internal projects at Alibaba, hessian2-codec is also used by the following notable projects:

  • The Envoy (Dubbo Filter in Envoy will use hessian2-codec as the serializer).

Contributing

Please read CONTRIBUTING.md for details on how to contribute to this project.

Happy testing!

Develop

Generate compile_commands.json for this repo by bazel run :refresh_compile_commands. Thank https://github.com/hedronvision/bazel-compile-commands-extractor for it provide the great script/tool to make this so easy!

License

hessian2-codec is distributed under Apache License 2.0.

Acknowledgements

邀请码
    Gitlink(确实开源)
  • 加入我们
  • 官网邮箱:gitlink@ccf.org.cn
  • QQ群
  • QQ群
  • 公众号
  • 公众号

版权所有:中国计算机学会技术支持:开源发展技术委员会
京ICP备13000930号-9 京公网安备 11010802032778号