core/xgo
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

some code clean up (#237)

Browse source
This commit is contained in:
daniel 2019-12-20 11:40:38 -08:00 committed by GitHub
parent 030b4b90ab
commit c88a5f506e
No known key found for this signature in database
GPG key ID: 4AEE18F83AFDEB23
297 changed files with 19913 additions and 19430 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

47
.circleci/config.yml Normal file
View file

@ -0,0 +1,47 @@
version: 2
jobs:
release:
docker:
- image: circleci/golang:1.13
working_directory: /go/src/github.com/d5/tegno
steps:
- checkout
- restore_cache:
keys:
- go-mod-v1-{{ checksum "go.sum" }}
- run: curl -sL https://git.io/goreleaser | bash
- save_cache:
key: go-mod-v1-{{ checksum "go.sum" }}
paths:
- "/go/pkg/mod"
test:
docker:
- image: circleci/golang:1.13
working_directory: /go/src/github.com/d5/tegno
steps:
- checkout
- restore_cache:
keys:
- go-mod-v1-{{ checksum "go.sum" }}
- run: go get -u golang.org/x/lint/golint
- run: make test
- save_cache:
key: go-mod-v1-{{ checksum "go.sum" }}
paths:
- "/go/pkg/mod"
workflows:
version: 2
test:
jobs:
- test:
filters:
tags:
ignore: /.*/
release:
jobs:
- release:
filters:
branches:
ignore: /.*/
tags:
only: /v[0-9]+(\.[0-9]+)*(-.*)*/

9
.goreleaser.yml
View file

@ -11,15 +11,6 @@ builds:
- darwin
- linux
- windows
- env:
- CGO_ENABLED=0
main: ./cmd/tengomin/main.go
id: tengomin
binary: tengomin
goos:
- darwin
- linux
- windows
archive:
files:
- none*

17
.travis.yml
View file

@ -1,17 +0,0 @@
language: go
go:
- "1.12"
install:
- env GO111MODULE=on go get -u golang.org/x/lint/golint
script:
- env GO111MODULE=on make test
deploy:
- provider: script
skip_cleanup: true
script: curl -sL https://git.io/goreleaser | bash
on:
tags: true

5
Makefile
View file

@ -1,13 +1,10 @@
vet:
go vet ./...
generate:
go generate ./...
lint:
golint -set_exit_status ./...
test: generate vet lint
test: generate lint
go test -race -cover ./...
fmt:

85
README.md
View file

@ -4,18 +4,18 @@
# The Tengo Language
[![GoDoc](https://godoc.org/github.com/d5/tengo?status.svg)](https://godoc.org/github.com/d5/tengo/script)
[![GoDoc](https://godoc.org/github.com/d5/tengo?status.svg)](https://godoc.org/github.com/d5/tengo)
[![Go Report Card](https://goreportcard.com/badge/github.com/d5/tengo)](https://goreportcard.com/report/github.com/d5/tengo)
[![Build Status](https://travis-ci.org/d5/tengo.svg?branch=master)](https://travis-ci.org/d5/tengo)
[![CircleCI](https://circleci.com/gh/d5/tengo.svg?style=svg)](https://circleci.com/gh/d5/tengo)
[![Sourcegraph](https://sourcegraph.com/github.com/d5/tengo/-/badge.svg)](https://sourcegraph.com/github.com/d5/tengo?badge)
**Tengo is a small, dynamic, fast, secure script language for Go.**
Tengo is **[fast](#benchmark)** and secure because it's compiled/executed as bytecode on stack-based VM that's written in native Go.
Tengo is **[fast](#benchmark)** and secure because it's compiled/executed as
bytecode on stack-based VM that's written in native Go.
```golang
/* The Tengo Language */
fmt := import("fmt")
each := func(seq, fn) {
@ -31,19 +31,24 @@ fmt.println(sum(0, [1, 2, 3])) // "6"
fmt.println(sum("", [1, 2, 3])) // "123"
```
> Run this code in the [Playground](https://tengolang.com/?s=0c8d5d0d88f2795a7093d7f35ae12c3afa17bea3)
> Test this Tengo code in the
> [Tengo Playground](https://tengolang.com/?s=0c8d5d0d88f2795a7093d7f35ae12c3afa17bea3)
## Features
- Simple and highly readable [Syntax](https://github.com/d5/tengo/blob/master/docs/tutorial.md)
- Simple and highly readable
[Syntax](https://github.com/d5/tengo/blob/master/docs/tutorial.md)
- Dynamic typing with type coercion
- Higher-order functions and closures
- Immutable values
- Garbage collection
- [Securely Embeddable](https://github.com/d5/tengo/blob/master/docs/interoperability.md) and [Extensible](https://github.com/d5/tengo/blob/master/docs/objects.md)
- [Securely Embeddable](https://github.com/d5/tengo/blob/master/docs/interoperability.md)
and [Extensible](https://github.com/d5/tengo/blob/master/docs/objects.md)
- Compiler/runtime written in native Go _(no external deps or cgo)_
- Executable as a [standalone](https://github.com/d5/tengo/blob/master/docs/tengo-cli.md) language / REPL
- Use cases: rules engine, [state machine](https://github.com/d5/go-fsm), [gaming](https://github.com/d5/pbr), data pipeline, [transpiler](https://github.com/d5/tengo2lua)
- Executable as a
[standalone](https://github.com/d5/tengo/blob/master/docs/tengo-cli.md)
language / REPL
- Use cases: rules engine, [state machine](https://github.com/d5/go-fsm),
data pipeline, [transpiler](https://github.com/d5/tengo2lua)
## Benchmark
@ -61,16 +66,70 @@ fmt.println(sum("", [1, 2, 3])) // "123"
| [otto](https://github.com/robertkrimen/otto) | `68,377ms` | `11ms` | JS Interpreter on Go |
| [Anko](https://github.com/mattn/anko) | `92,579ms` | `18ms` | Interpreter on Go |
_* [fib(35)](https://github.com/d5/tengobench/blob/master/code/fib.tengo): Fibonacci(35)_
_* [fibt(35)](https://github.com/d5/tengobench/blob/master/code/fibtc.tengo): [tail-call](https://en.wikipedia.org/wiki/Tail_call) version of Fibonacci(35)_
_* [fib(35)](https://github.com/d5/tengobench/blob/master/code/fib.tengo):
Fibonacci(35)_
_* [fibt(35)](https://github.com/d5/tengobench/blob/master/code/fibtc.tengo):
[tail-call](https://en.wikipedia.org/wiki/Tail_call) version of Fibonacci(35)_
_* **Go** does not read the source code from file, while all other cases do_
_* See [here](https://github.com/d5/tengobench) for commands/codes used_
## Quick Start
A simple Go example code that compiles/runs Tengo script code with some input/output values:
```golang
package main
import (
"context"
"fmt"
"github.com/d5/tengo"
)
func main() {
// Tengo script code
src := `
each := func(seq, fn) {
for x in seq { fn(x) }
}
sum := 0
mul := 1
each([a, b, c, d], func(x) {
sum += x
mul *= x
})`
// create a new Script instance
script := tengo.NewScript([]byte(src))
// set values
_ = script.Add("a", 1)
_ = script.Add("b", 9)
_ = script.Add("c", 8)
_ = script.Add("d", 4)
// run the script
compiled, err := script.RunContext(context.Background())
if err != nil {
panic(err)
}
// retrieve values
sum := compiled.Get("sum")
mul := compiled.Get("mul")
fmt.Println(sum, mul) // "22 288"
}
```
## References
- [Language Syntax](https://github.com/d5/tengo/blob/master/docs/tutorial.md)
- [Object Types](https://github.com/d5/tengo/blob/master/docs/objects.md)
- [Runtime Types](https://github.com/d5/tengo/blob/master/docs/runtime-types.md) and [Operators](https://github.com/d5/tengo/blob/master/docs/operators.md)
- [Runtime Types](https://github.com/d5/tengo/blob/master/docs/runtime-types.md)
and [Operators](https://github.com/d5/tengo/blob/master/docs/operators.md)
- [Builtin Functions](https://github.com/d5/tengo/blob/master/docs/builtins.md)
- [Interoperability](https://github.com/d5/tengo/blob/master/docs/interoperability.md)
- [Tengo CLI](https://github.com/d5/tengo/blob/master/docs/tengo-cli.md)

356
assert/assert.go
View file

@ -1,356 +0,0 @@
package assert
import (
"bytes"
"fmt"
"reflect"
"strings"
"testing"
"github.com/d5/tengo/compiler"
"github.com/d5/tengo/compiler/source"
"github.com/d5/tengo/compiler/token"
"github.com/d5/tengo/objects"
)
// NoError asserts err is not an error.
func NoError(t *testing.T, err error, msg ...interface{}) bool {
if err == nil {
return true
}
return failExpectedActual(t, "no error", err, msg...)
}
// Error asserts err is an error.
func Error(t *testing.T, err error, msg ...interface{}) bool {
if err != nil {
return true
}
return failExpectedActual(t, "error", err, msg...)
}
// Nil asserts v is nil.
func Nil(t *testing.T, v interface{}, msg ...interface{}) bool {
if isNil(v) {
return true
}
return failExpectedActual(t, "nil", v, msg...)
}
// True asserts v is true.
func True(t *testing.T, v bool, msg ...interface{}) bool {
if v {
return true
}
return failExpectedActual(t, "true", v, msg...)
}
// False asserts vis false.
func False(t *testing.T, v bool, msg ...interface{}) bool {
if !v {
return true
}
return failExpectedActual(t, "false", v, msg...)
}
// NotNil asserts v is not nil.
func NotNil(t *testing.T, v interface{}, msg ...interface{}) bool {
if !isNil(v) {
return true
}
return failExpectedActual(t, "not nil", v, msg...)
}
// IsType asserts expected and actual are of the same type.
func IsType(t *testing.T, expected, actual interface{}, msg ...interface{}) bool {
if reflect.TypeOf(expected) == reflect.TypeOf(actual) {
return true
}
return failExpectedActual(t, reflect.TypeOf(expected), reflect.TypeOf(actual), msg...)
}
// Equal asserts expected and actual are equal.
func Equal(t *testing.T, expected, actual interface{}, msg ...interface{}) bool {
if isNil(expected) {
return Nil(t, actual, "expected nil, but got not nil")
}
if !NotNil(t, actual, "expected not nil, but got nil") {
return false
}
if !IsType(t, expected, actual, msg...) {
return false
}
switch expected := expected.(type) {
case int:
if expected != actual.(int) {
return failExpectedActual(t, expected, actual, msg...)
}
case int64:
if expected != actual.(int64) {
return failExpectedActual(t, expected, actual, msg...)
}
case float64:
if expected != actual.(float64) {
return failExpectedActual(t, expected, actual, msg...)
}
case string:
if expected != actual.(string) {
return failExpectedActual(t, expected, actual, msg...)
}
case []byte:
if !bytes.Equal(expected, actual.([]byte)) {
return failExpectedActual(t, string(expected), string(actual.([]byte)), msg...)
}
case []string:
if !equalStringSlice(expected, actual.([]string)) {
return failExpectedActual(t, expected, actual, msg...)
}
case []int:
if !equalIntSlice(expected, actual.([]int)) {
return failExpectedActual(t, expected, actual, msg...)
}
case bool:
if expected != actual.(bool) {
return failExpectedActual(t, expected, actual, msg...)
}
case rune:
if expected != actual.(rune) {
return failExpectedActual(t, expected, actual, msg...)
}
case *compiler.Symbol:
if !equalSymbol(expected, actual.(*compiler.Symbol)) {
return failExpectedActual(t, expected, actual, msg...)
}
case source.Pos:
if expected != actual.(source.Pos) {
return failExpectedActual(t, expected, actual, msg...)
}
case token.Token:
if expected != actual.(token.Token) {
return failExpectedActual(t, expected, actual, msg...)
}
case []objects.Object:
return equalObjectSlice(t, expected, actual.([]objects.Object), msg...)
case *objects.Int:
return Equal(t, expected.Value, actual.(*objects.Int).Value, msg...)
case *objects.Float:
return Equal(t, expected.Value, actual.(*objects.Float).Value, msg...)
case *objects.String:
return Equal(t, expected.Value, actual.(*objects.String).Value, msg...)
case *objects.Char:
return Equal(t, expected.Value, actual.(*objects.Char).Value, msg...)
case *objects.Bool:
if expected != actual {
return failExpectedActual(t, expected, actual, msg...)
}
case *objects.Array:
return equalObjectSlice(t, expected.Value, actual.(*objects.Array).Value, msg...)
case *objects.ImmutableArray:
return equalObjectSlice(t, expected.Value, actual.(*objects.ImmutableArray).Value, msg...)
case *objects.Bytes:
if !bytes.Equal(expected.Value, actual.(*objects.Bytes).Value) {
return failExpectedActual(t, string(expected.Value), string(actual.(*objects.Bytes).Value), msg...)
}
case *objects.Map:
return equalObjectMap(t, expected.Value, actual.(*objects.Map).Value, msg...)
case *objects.ImmutableMap:
return equalObjectMap(t, expected.Value, actual.(*objects.ImmutableMap).Value, msg...)
case *objects.CompiledFunction:
return equalCompiledFunction(t, expected, actual.(*objects.CompiledFunction), msg...)
case *objects.Closure:
return equalClosure(t, expected, actual.(*objects.Closure), msg...)
case *objects.Undefined:
if expected != actual {
return failExpectedActual(t, expected, actual, msg...)
}
case *objects.Error:
return Equal(t, expected.Value, actual.(*objects.Error).Value, msg...)
case objects.Object:
if !expected.Equals(actual.(objects.Object)) {
return failExpectedActual(t, expected, actual, msg...)
}
case *source.FileSet:
return equalFileSet(t, expected, actual.(*source.FileSet), msg...)
case *source.File:
return Equal(t, expected.Name, actual.(*source.File).Name, msg...) &&
Equal(t, expected.Base, actual.(*source.File).Base, msg...) &&
Equal(t, expected.Size, actual.(*source.File).Size, msg...) &&
True(t, equalIntSlice(expected.Lines, actual.(*source.File).Lines), msg...)
case error:
if expected != actual.(error) {
return failExpectedActual(t, expected, actual, msg...)
}
default:
panic(fmt.Errorf("type not implemented: %T", expected))
}
return true
}
// Fail marks the function as having failed but continues execution.
func Fail(t *testing.T, msg ...interface{}) bool {
t.Logf("\nError trace:\n\t%s\n%s", strings.Join(errorTrace(), "\n\t"), message(msg...))
t.Fail()
return false
}
func failExpectedActual(t *testing.T, expected, actual interface{}, msg ...interface{}) bool {
var addMsg string
if len(msg) > 0 {
addMsg = "\nMessage: " + message(msg...)
}
t.Logf("\nError trace:\n\t%s\nExpected: %v\nActual: %v%s",
strings.Join(errorTrace(), "\n\t"),
expected, actual,
addMsg)
t.Fail()
return false
}
func message(formatArgs ...interface{}) string {
var format string
var args []interface{}
if len(formatArgs) > 0 {
format = formatArgs[0].(string)
}
if len(formatArgs) > 1 {
args = formatArgs[1:]
}
return fmt.Sprintf(format, args...)
}
func equalIntSlice(a, b []int) bool {
if len(a) != len(b) {
return false
}
for i := 0; i < len(a); i++ {
if a[i] != b[i] {
return false
}
}
return true
}
func equalStringSlice(a, b []string) bool {
if len(a) != len(b) {
return false
}
for i := 0; i < len(a); i++ {
if a[i] != b[i] {
return false
}
}
return true
}
func equalSymbol(a, b *compiler.Symbol) bool {
return a.Name == b.Name &&
a.Index == b.Index &&
a.Scope == b.Scope
}
func equalObjectSlice(t *testing.T, expected, actual []objects.Object, msg ...interface{}) bool {
if !Equal(t, len(expected), len(actual), msg...) {
return false
}
for i := 0; i < len(expected); i++ {
if !Equal(t, expected[i], actual[i], msg...) {
return false
}
}
return true
}
func equalFileSet(t *testing.T, expected, actual *source.FileSet, msg ...interface{}) bool {
if !Equal(t, len(expected.Files), len(actual.Files), msg...) {
return false
}
for i, f := range expected.Files {
if !Equal(t, f, actual.Files[i], msg...) {
return false
}
}
return Equal(t, expected.Base, actual.Base) &&
Equal(t, expected.LastFile, actual.LastFile)
}
func equalObjectMap(t *testing.T, expected, actual map[string]objects.Object, msg ...interface{}) bool {
if !Equal(t, len(expected), len(actual), msg...) {
return false
}
for key, expectedVal := range expected {
actualVal := actual[key]
if !Equal(t, expectedVal, actualVal, msg...) {
return false
}
}
return true
}
func equalCompiledFunction(t *testing.T, expected, actual objects.Object, msg ...interface{}) bool {
expectedT := expected.(*objects.CompiledFunction)
actualT := actual.(*objects.CompiledFunction)
return Equal(t,
compiler.FormatInstructions(expectedT.Instructions, 0),
compiler.FormatInstructions(actualT.Instructions, 0), msg...)
}
func equalClosure(t *testing.T, expected, actual objects.Object, msg ...interface{}) bool {
expectedT := expected.(*objects.Closure)
actualT := actual.(*objects.Closure)
if !Equal(t, expectedT.Fn, actualT.Fn, msg...) {
return false
}
if !Equal(t, len(expectedT.Free), len(actualT.Free), msg...) {
return false
}
for i := 0; i < len(expectedT.Free); i++ {
if !Equal(t, *expectedT.Free[i], *actualT.Free[i], msg...) {
return false
}
}
return true
}
func isNil(v interface{}) bool {
if v == nil {
return true
}
value := reflect.ValueOf(v)
kind := value.Kind()
if kind >= reflect.Chan && kind <= reflect.Slice && value.IsNil() {
return true
}
return false
}

72
assert/trace.go
View file

@ -1,72 +0,0 @@
package assert
import (
"fmt"
"runtime"
"strings"
"unicode"
"unicode/utf8"
)
func errorTrace() []string {
var pc uintptr
file := ""
line := 0
var ok bool
name := ""
var callers []string
for i := 0; ; i++ {
pc, file, line, ok = runtime.Caller(i)
if !ok {
break
}
if file == "<autogenerated>" {
break
}
f := runtime.FuncForPC(pc)
if f == nil {
break
}
name = f.Name()
if name == "testing.tRunner" {
break
}
parts := strings.Split(file, "/")
file = parts[len(parts)-1]
if len(parts) > 1 {
dir := parts[len(parts)-2]
if (dir != "assert" && dir != "mock" && dir != "require") || file == "mock_test.go" {
callers = append(callers, fmt.Sprintf("%s:%d", file, line))
}
}
// Drop the package
segments := strings.Split(name, ".")
name = segments[len(segments)-1]
if isTest(name, "Test") ||
isTest(name, "Benchmark") ||
isTest(name, "Example") {
break
}
}
return callers
}
func isTest(name, prefix string) bool {
if !strings.HasPrefix(name, prefix) {
return false
}
if len(name) == len(prefix) { // "Test" is ok
return true
}
r, _ := utf8.DecodeRuneInString(name[len(prefix):])
return !unicode.IsLower(r)
}

502
builtins.go Normal file
View file

@ -0,0 +1,502 @@
package tengo
var builtinFuncs = []*BuiltinFunction{
{
Name: "len",
Value: builtinLen,
},
{
Name: "copy",
Value: builtinCopy,
},
{
Name: "append",
Value: builtinAppend,
},
{
Name: "string",
Value: builtinString,
},
{
Name: "int",
Value: builtinInt,
},
{
Name: "bool",
Value: builtinBool,
},
{
Name: "float",
Value: builtinFloat,
},
{
Name: "char",
Value: builtinChar,
},
{
Name: "bytes",
Value: builtinBytes,
},
{
Name: "time",
Value: builtinTime,
},
{
Name: "is_int",
Value: builtinIsInt,
},
{
Name: "is_float",
Value: builtinIsFloat,
},
{
Name: "is_string",
Value: builtinIsString,
},
{
Name: "is_bool",
Value: builtinIsBool,
},
{
Name: "is_char",
Value: builtinIsChar,
},
{
Name: "is_bytes",
Value: builtinIsBytes,
},
{
Name: "is_array",
Value: builtinIsArray,
},
{
Name: "is_immutable_array",
Value: builtinIsImmutableArray,
},
{
Name: "is_map",
Value: builtinIsMap,
},
{
Name: "is_immutable_map",
Value: builtinIsImmutableMap,
},
{
Name: "is_iterable",
Value: builtinIsIterable,
},
{
Name: "is_time",
Value: builtinIsTime,
},
{
Name: "is_error",
Value: builtinIsError,
},
{
Name: "is_undefined",
Value: builtinIsUndefined,
},
{
Name: "is_function",
Value: builtinIsFunction,
},
{
Name: "is_callable",
Value: builtinIsCallable,
},
{
Name: "type_name",
Value: builtinTypeName,
},
{
Name: "format",
Value: builtinFormat,
},
}
// GetAllBuiltinFunctions returns all builtin function objects.
func GetAllBuiltinFunctions() []*BuiltinFunction {
return append([]*BuiltinFunction{}, builtinFuncs...)
}
func builtinTypeName(args ...Object) (Object, error) {
if len(args) != 1 {
return nil, ErrWrongNumArguments
}
return &String{Value: args[0].TypeName()}, nil
}
func builtinIsString(args ...Object) (Object, error) {
if len(args) != 1 {
return nil, ErrWrongNumArguments
}
if _, ok := args[0].(*String); ok {
return TrueValue, nil
}
return FalseValue, nil
}
func builtinIsInt(args ...Object) (Object, error) {
if len(args) != 1 {
return nil, ErrWrongNumArguments
}
if _, ok := args[0].(*Int); ok {
return TrueValue, nil
}
return FalseValue, nil
}
func builtinIsFloat(args ...Object) (Object, error) {
if len(args) != 1 {
return nil, ErrWrongNumArguments
}
if _, ok := args[0].(*Float); ok {
return TrueValue, nil
}
return FalseValue, nil
}
func builtinIsBool(args ...Object) (Object, error) {
if len(args) != 1 {
return nil, ErrWrongNumArguments
}
if _, ok := args[0].(*Bool); ok {
return TrueValue, nil
}
return FalseValue, nil
}
func builtinIsChar(args ...Object) (Object, error) {
if len(args) != 1 {
return nil, ErrWrongNumArguments
}
if _, ok := args[0].(*Char); ok {
return TrueValue, nil
}
return FalseValue, nil
}
func builtinIsBytes(args ...Object) (Object, error) {
if len(args) != 1 {
return nil, ErrWrongNumArguments
}
if _, ok := args[0].(*Bytes); ok {
return TrueValue, nil
}
return FalseValue, nil
}
func builtinIsArray(args ...Object) (Object, error) {
if len(args) != 1 {
return nil, ErrWrongNumArguments
}
if _, ok := args[0].(*Array); ok {
return TrueValue, nil
}
return FalseValue, nil
}
func builtinIsImmutableArray(args ...Object) (Object, error) {
if len(args) != 1 {
return nil, ErrWrongNumArguments
}
if _, ok := args[0].(*ImmutableArray); ok {
return TrueValue, nil
}
return FalseValue, nil
}
func builtinIsMap(args ...Object) (Object, error) {
if len(args) != 1 {
return nil, ErrWrongNumArguments
}
if _, ok := args[0].(*Map); ok {
return TrueValue, nil
}
return FalseValue, nil
}
func builtinIsImmutableMap(args ...Object) (Object, error) {
if len(args) != 1 {
return nil, ErrWrongNumArguments
}
if _, ok := args[0].(*ImmutableMap); ok {
return TrueValue, nil
}
return FalseValue, nil
}
func builtinIsTime(args ...Object) (Object, error) {
if len(args) != 1 {
return nil, ErrWrongNumArguments
}
if _, ok := args[0].(*Time); ok {
return TrueValue, nil
}
return FalseValue, nil
}
func builtinIsError(args ...Object) (Object, error) {
if len(args) != 1 {
return nil, ErrWrongNumArguments
}
if _, ok := args[0].(*Error); ok {
return TrueValue, nil
}
return FalseValue, nil
}
func builtinIsUndefined(args ...Object) (Object, error) {
if len(args) != 1 {
return nil, ErrWrongNumArguments
}
if args[0] == UndefinedValue {
return TrueValue, nil
}
return FalseValue, nil
}
func builtinIsFunction(args ...Object) (Object, error) {
if len(args) != 1 {
return nil, ErrWrongNumArguments
}
switch args[0].(type) {
case *CompiledFunction:
return TrueValue, nil
}
return FalseValue, nil
}
func builtinIsCallable(args ...Object) (Object, error) {
if len(args) != 1 {
return nil, ErrWrongNumArguments
}
if args[0].CanCall() {
return TrueValue, nil
}
return FalseValue, nil
}
func builtinIsIterable(args ...Object) (Object, error) {
if len(args) != 1 {
return nil, ErrWrongNumArguments
}
if args[0].CanIterate() {
return TrueValue, nil
}
return FalseValue, nil
}
// len(obj object) => int
func builtinLen(args ...Object) (Object, error) {
if len(args) != 1 {
return nil, ErrWrongNumArguments
}
switch arg := args[0].(type) {
case *Array:
return &Int{Value: int64(len(arg.Value))}, nil
case *ImmutableArray:
return &Int{Value: int64(len(arg.Value))}, nil
case *String:
return &Int{Value: int64(len(arg.Value))}, nil
case *Bytes:
return &Int{Value: int64(len(arg.Value))}, nil
case *Map:
return &Int{Value: int64(len(arg.Value))}, nil
case *ImmutableMap:
return &Int{Value: int64(len(arg.Value))}, nil
default:
return nil, ErrInvalidArgumentType{
Name: "first",
Expected: "array/string/bytes/map",
Found: arg.TypeName(),
}
}
}
func builtinFormat(args ...Object) (Object, error) {
numArgs := len(args)
if numArgs == 0 {
return nil, ErrWrongNumArguments
}
format, ok := args[0].(*String)
if !ok {
return nil, ErrInvalidArgumentType{
Name: "format",
Expected: "string",
Found: args[0].TypeName(),
}
}
if numArgs == 1 {
// okay to return 'format' directly as String is immutable
return format, nil
}
s, err := Format(format.Value, args[1:]...)
if err != nil {
return nil, err
}
return &String{Value: s}, nil
}
func builtinCopy(args ...Object) (Object, error) {
if len(args) != 1 {
return nil, ErrWrongNumArguments
}
return args[0].Copy(), nil
}
func builtinString(args ...Object) (Object, error) {
argsLen := len(args)
if !(argsLen == 1 || argsLen == 2) {
return nil, ErrWrongNumArguments
}
if _, ok := args[0].(*String); ok {
return args[0], nil
}
v, ok := ToString(args[0])
if ok {
if len(v) > MaxStringLen {
return nil, ErrStringLimit
}
return &String{Value: v}, nil
}
if argsLen == 2 {
return args[1], nil
}
return UndefinedValue, nil
}
func builtinInt(args ...Object) (Object, error) {
argsLen := len(args)
if !(argsLen == 1 || argsLen == 2) {
return nil, ErrWrongNumArguments
}
if _, ok := args[0].(*Int); ok {
return args[0], nil
}
v, ok := ToInt64(args[0])
if ok {
return &Int{Value: v}, nil
}
if argsLen == 2 {
return args[1], nil
}
return UndefinedValue, nil
}
func builtinFloat(args ...Object) (Object, error) {
argsLen := len(args)
if !(argsLen == 1 || argsLen == 2) {
return nil, ErrWrongNumArguments
}
if _, ok := args[0].(*Float); ok {
return args[0], nil
}
v, ok := ToFloat64(args[0])
if ok {
return &Float{Value: v}, nil
}
if argsLen == 2 {
return args[1], nil
}
return UndefinedValue, nil
}
func builtinBool(args ...Object) (Object, error) {
if len(args) != 1 {
return nil, ErrWrongNumArguments
}
if _, ok := args[0].(*Bool); ok {
return args[0], nil
}
v, ok := ToBool(args[0])
if ok {
if v {
return TrueValue, nil
}
return FalseValue, nil
}
return UndefinedValue, nil
}
func builtinChar(args ...Object) (Object, error) {
argsLen := len(args)
if !(argsLen == 1 || argsLen == 2) {
return nil, ErrWrongNumArguments
}
if _, ok := args[0].(*Char); ok {
return args[0], nil
}
v, ok := ToRune(args[0])
if ok {
return &Char{Value: v}, nil
}
if argsLen == 2 {
return args[1], nil
}
return UndefinedValue, nil
}
func builtinBytes(args ...Object) (Object, error) {
argsLen := len(args)
if !(argsLen == 1 || argsLen == 2) {
return nil, ErrWrongNumArguments
}
// bytes(N) => create a new bytes with given size N
if n, ok := args[0].(*Int); ok {
if n.Value > int64(MaxBytesLen) {
return nil, ErrBytesLimit
}
return &Bytes{Value: make([]byte, int(n.Value))}, nil
}
v, ok := ToByteSlice(args[0])
if ok {
if len(v) > MaxBytesLen {
return nil, ErrBytesLimit
}
return &Bytes{Value: v}, nil
}
if argsLen == 2 {
return args[1], nil
}
return UndefinedValue, nil
}
func builtinTime(args ...Object) (Object, error) {
argsLen := len(args)
if !(argsLen == 1 || argsLen == 2) {
return nil, ErrWrongNumArguments
}
if _, ok := args[0].(*Time); ok {
return args[0], nil
}
v, ok := ToTime(args[0])
if ok {
return &Time{Value: v}, nil
}
if argsLen == 2 {
return args[1], nil
}
return UndefinedValue, nil
}
// append(arr, items...)
func builtinAppend(args ...Object) (Object, error) {
if len(args) < 2 {
return nil, ErrWrongNumArguments
}
switch arg := args[0].(type) {
case *Array:
return &Array{Value: append(arg.Value, args[1:]...)}, nil
case *ImmutableArray:
return &Array{Value: append(arg.Value, args[1:]...)}, nil
default:
return nil, ErrInvalidArgumentType{
Name: "first",
Expected: "array",
Found: arg.TypeName(),
}
}
}

292
bytecode.go Normal file
View file

@ -0,0 +1,292 @@
package tengo
import (
"encoding/gob"
"fmt"
"io"
"reflect"
"github.com/d5/tengo/internal"
)
// Bytecode is a compiled instructions and constants.
type Bytecode struct {
FileSet *internal.SourceFileSet
MainFunction *CompiledFunction
Constants []Object
}
// Encode writes Bytecode data to the writer.
func (b *Bytecode) Encode(w io.Writer) error {
enc := gob.NewEncoder(w)
if err := enc.Encode(b.FileSet); err != nil {
return err
}
if err := enc.Encode(b.MainFunction); err != nil {
return err
}
return enc.Encode(b.Constants)
}
// CountObjects returns the number of objects found in Constants.
func (b *Bytecode) CountObjects() int {
n := 0
for _, c := range b.Constants {
n += CountObjects(c)
}
return n
}
// FormatInstructions returns human readable string representations of
// compiled instructions.
func (b *Bytecode) FormatInstructions() []string {
return internal.FormatInstructions(b.MainFunction.Instructions, 0)
}
// FormatConstants returns human readable string representations of
// compiled constants.
func (b *Bytecode) FormatConstants() (output []string) {
for cidx, cn := range b.Constants {
switch cn := cn.(type) {
case *CompiledFunction:
output = append(output, fmt.Sprintf(
"[% 3d] (Compiled Function|%p)", cidx, &cn))
for _, l := range internal.FormatInstructions(cn.Instructions, 0) {
output = append(output, fmt.Sprintf(" %s", l))
}
default:
output = append(output, fmt.Sprintf("[% 3d] %s (%s|%p)",
cidx, cn, reflect.TypeOf(cn).Elem().Name(), &cn))
}
}
return
}
// Decode reads Bytecode data from the reader.
func (b *Bytecode) Decode(r io.Reader, modules *ModuleMap) error {
if modules == nil {
modules = NewModuleMap()
}
dec := gob.NewDecoder(r)
if err := dec.Decode(&b.FileSet); err != nil {
return err
}
// TODO: files in b.FileSet.File does not have their 'set' field properly
// set to b.FileSet as it's private field and not serialized by gob
// encoder/decoder.
if err := dec.Decode(&b.MainFunction); err != nil {
return err
}
if err := dec.Decode(&b.Constants); err != nil {
return err
}
for i, v := range b.Constants {
fv, err := fixDecodedObject(v, modules)
if err != nil {
return err
}
b.Constants[i] = fv
}
return nil
}
// RemoveDuplicates finds and remove the duplicate values in Constants.
// Note this function mutates Bytecode.
func (b *Bytecode) RemoveDuplicates() {
var deduped []Object
indexMap := make(map[int]int) // mapping from old constant index to new index
ints := make(map[int64]int)
strings := make(map[string]int)
floats := make(map[float64]int)
chars := make(map[rune]int)
immutableMaps := make(map[string]int) // for modules
for curIdx, c := range b.Constants {
switch c := c.(type) {
case *CompiledFunction:
// add to deduped list
indexMap[curIdx] = len(deduped)
deduped = append(deduped, c)
case *ImmutableMap:
modName := inferModuleName(c)
newIdx, ok := immutableMaps[modName]
if modName != "" && ok {
indexMap[curIdx] = newIdx
} else {
newIdx = len(deduped)
immutableMaps[modName] = newIdx
indexMap[curIdx] = newIdx
deduped = append(deduped, c)
}
case *Int:
if newIdx, ok := ints[c.Value]; ok {
indexMap[curIdx] = newIdx
} else {
newIdx = len(deduped)
ints[c.Value] = newIdx
indexMap[curIdx] = newIdx
deduped = append(deduped, c)
}
case *String:
if newIdx, ok := strings[c.Value]; ok {
indexMap[curIdx] = newIdx
} else {
newIdx = len(deduped)
strings[c.Value] = newIdx
indexMap[curIdx] = newIdx
deduped = append(deduped, c)
}
case *Float:
if newIdx, ok := floats[c.Value]; ok {
indexMap[curIdx] = newIdx
} else {
newIdx = len(deduped)
floats[c.Value] = newIdx
indexMap[curIdx] = newIdx
deduped = append(deduped, c)
}
case *Char:
if newIdx, ok := chars[c.Value]; ok {
indexMap[curIdx] = newIdx
} else {
newIdx = len(deduped)
chars[c.Value] = newIdx
indexMap[curIdx] = newIdx
deduped = append(deduped, c)
}
default:
panic(fmt.Errorf("unsupported top-level constant type: %s",
c.TypeName()))
}
}
// replace with de-duplicated constants
b.Constants = deduped
// update CONST instructions with new indexes
// main function
updateConstIndexes(b.MainFunction.Instructions, indexMap)
// other compiled functions in constants
for _, c := range b.Constants {
switch c := c.(type) {
case *CompiledFunction:
updateConstIndexes(c.Instructions, indexMap)
}
}
}
func fixDecodedObject(
o Object,
modules *ModuleMap,
) (Object, error) {
switch o := o.(type) {
case *Bool:
if o.IsFalsy() {
return FalseValue, nil
}
return TrueValue, nil
case *Undefined:
return UndefinedValue, nil
case *Array:
for i, v := range o.Value {
fv, err := fixDecodedObject(v, modules)
if err != nil {
return nil, err
}
o.Value[i] = fv
}
case *ImmutableArray:
for i, v := range o.Value {
fv, err := fixDecodedObject(v, modules)
if err != nil {
return nil, err
}
o.Value[i] = fv
}
case *Map:
for k, v := range o.Value {
fv, err := fixDecodedObject(v, modules)
if err != nil {
return nil, err
}
o.Value[k] = fv
}
case *ImmutableMap:
modName := inferModuleName(o)
if mod := modules.GetBuiltinModule(modName); mod != nil {
return mod.AsImmutableMap(modName), nil
}
for k, v := range o.Value {
// encoding of user function not supported
if _, isUserFunction := v.(*UserFunction); isUserFunction {
return nil, fmt.Errorf("user function not decodable")
}
fv, err := fixDecodedObject(v, modules)
if err != nil {
return nil, err
}
o.Value[k] = fv
}
}
return o, nil
}
func updateConstIndexes(insts []byte, indexMap map[int]int) {
i := 0
for i < len(insts) {
op := insts[i]
numOperands := internal.OpcodeOperands[op]
_, read := internal.ReadOperands(numOperands, insts[i+1:])
switch op {
case internal.OpConstant:
curIdx := int(insts[i+2]) | int(insts[i+1])<<8
newIdx, ok := indexMap[curIdx]
if !ok {
panic(fmt.Errorf("constant index not found: %d", curIdx))
}
copy(insts[i:], internal.MakeInstruction(op, newIdx))
case internal.OpClosure:
curIdx := int(insts[i+2]) | int(insts[i+1])<<8
numFree := int(insts[i+3])
newIdx, ok := indexMap[curIdx]
if !ok {
panic(fmt.Errorf("constant index not found: %d", curIdx))
}
copy(insts[i:], internal.MakeInstruction(op, newIdx, numFree))
}
i += 1 + read
}
}
func inferModuleName(mod *ImmutableMap) string {
if modName, ok := mod.Value["__module_name__"].(*String); ok {
return modName.Value
}
return ""
}
func init() {
gob.Register(&internal.SourceFileSet{})
gob.Register(&internal.SourceFile{})
gob.Register(&Array{})
gob.Register(&Bool{})
gob.Register(&Bytes{})
gob.Register(&Char{})
gob.Register(&CompiledFunction{})
gob.Register(&Error{})
gob.Register(&Float{})
gob.Register(&ImmutableArray{})
gob.Register(&ImmutableMap{})
gob.Register(&Int{})
gob.Register(&Map{})
gob.Register(&String{})
gob.Register(&Time{})
gob.Register(&Undefined{})
gob.Register(&UserFunction{})
}

298
bytecode_test.go Normal file
View file

@ -0,0 +1,298 @@
package tengo_test
import (
"bytes"
"testing"
"time"
"github.com/d5/tengo"
"github.com/d5/tengo/internal"
"github.com/d5/tengo/internal/require"
)
type srcfile struct {
name string
size int
}
func TestBytecode(t *testing.T) {
testBytecodeSerialization(t, bytecode(concatInsts(), objectsArray()))
testBytecodeSerialization(t, bytecode(
concatInsts(), objectsArray(
&tengo.Char{Value: 'y'},
&tengo.Float{Value: 93.11},
compiledFunction(1, 0,
internal.MakeInstruction(internal.OpConstant, 3),
internal.MakeInstruction(internal.OpSetLocal, 0),
internal.MakeInstruction(internal.OpGetGlobal, 0),
internal.MakeInstruction(internal.OpGetFree, 0)),
&tengo.Float{Value: 39.2},
&tengo.Int{Value: 192},
&tengo.String{Value: "bar"})))
testBytecodeSerialization(t, bytecodeFileSet(
concatInsts(
internal.MakeInstruction(internal.OpConstant, 0),
internal.MakeInstruction(internal.OpSetGlobal, 0),
internal.MakeInstruction(internal.OpConstant, 6),
internal.MakeInstruction(internal.OpPop)),
objectsArray(
&tengo.Int{Value: 55},
&tengo.Int{Value: 66},
&tengo.Int{Value: 77},
&tengo.Int{Value: 88},
&tengo.ImmutableMap{
Value: map[string]tengo.Object{
"array": &tengo.ImmutableArray{
Value: []tengo.Object{
&tengo.Int{Value: 1},
&tengo.Int{Value: 2},
&tengo.Int{Value: 3},
tengo.TrueValue,
tengo.FalseValue,
tengo.UndefinedValue,
},
},
"true": tengo.TrueValue,
"false": tengo.FalseValue,
"bytes": &tengo.Bytes{Value: make([]byte, 16)},
"char": &tengo.Char{Value: 'Y'},
"error": &tengo.Error{Value: &tengo.String{
Value: "some error",
}},
"float": &tengo.Float{Value: -19.84},
"immutable_array": &tengo.ImmutableArray{
Value: []tengo.Object{
&tengo.Int{Value: 1},
&tengo.Int{Value: 2},
&tengo.Int{Value: 3},
tengo.TrueValue,
tengo.FalseValue,
tengo.UndefinedValue,
},
},
"immutable_map": &tengo.ImmutableMap{
Value: map[string]tengo.Object{
"a": &tengo.Int{Value: 1},
"b": &tengo.Int{Value: 2},
"c": &tengo.Int{Value: 3},
"d": tengo.TrueValue,
"e": tengo.FalseValue,
"f": tengo.UndefinedValue,
},
},
"int": &tengo.Int{Value: 91},
"map": &tengo.Map{
Value: map[string]tengo.Object{
"a": &tengo.Int{Value: 1},
"b": &tengo.Int{Value: 2},
"c": &tengo.Int{Value: 3},
"d": tengo.TrueValue,
"e": tengo.FalseValue,
"f": tengo.UndefinedValue,
},
},
"string": &tengo.String{Value: "foo bar"},
"time": &tengo.Time{Value: time.Now()},
"undefined": tengo.UndefinedValue,
},
},
compiledFunction(1, 0,
internal.MakeInstruction(internal.OpConstant, 3),
internal.MakeInstruction(internal.OpSetLocal, 0),
internal.MakeInstruction(internal.OpGetGlobal, 0),
internal.MakeInstruction(internal.OpGetFree, 0),
internal.MakeInstruction(internal.OpBinaryOp, 11),
internal.MakeInstruction(internal.OpGetFree, 1),
internal.MakeInstruction(internal.OpBinaryOp, 11),
internal.MakeInstruction(internal.OpGetLocal, 0),
internal.MakeInstruction(internal.OpBinaryOp, 11),
internal.MakeInstruction(internal.OpReturn, 1)),
compiledFunction(1, 0,
internal.MakeInstruction(internal.OpConstant, 2),
internal.MakeInstruction(internal.OpSetLocal, 0),
internal.MakeInstruction(internal.OpGetFree, 0),
internal.MakeInstruction(internal.OpGetLocal, 0),
internal.MakeInstruction(internal.OpClosure, 4, 2),
internal.MakeInstruction(internal.OpReturn, 1)),
compiledFunction(1, 0,
internal.MakeInstruction(internal.OpConstant, 1),
internal.MakeInstruction(internal.OpSetLocal, 0),
internal.MakeInstruction(internal.OpGetLocal, 0),
internal.MakeInstruction(internal.OpClosure, 5, 1),
internal.MakeInstruction(internal.OpReturn, 1))),
fileSet(srcfile{name: "file1", size: 100},
srcfile{name: "file2", size: 200})))
}
func TestBytecode_RemoveDuplicates(t *testing.T) {
testBytecodeRemoveDuplicates(t,
bytecode(
concatInsts(), objectsArray(
&tengo.Char{Value: 'y'},
&tengo.Float{Value: 93.11},
compiledFunction(1, 0,
internal.MakeInstruction(internal.OpConstant, 3),
internal.MakeInstruction(internal.OpSetLocal, 0),
internal.MakeInstruction(internal.OpGetGlobal, 0),
internal.MakeInstruction(internal.OpGetFree, 0)),
&tengo.Float{Value: 39.2},
&tengo.Int{Value: 192},
&tengo.String{Value: "bar"})),
bytecode(
concatInsts(), objectsArray(
&tengo.Char{Value: 'y'},
&tengo.Float{Value: 93.11},
compiledFunction(1, 0,
internal.MakeInstruction(internal.OpConstant, 3),
internal.MakeInstruction(internal.OpSetLocal, 0),
internal.MakeInstruction(internal.OpGetGlobal, 0),
internal.MakeInstruction(internal.OpGetFree, 0)),
&tengo.Float{Value: 39.2},
&tengo.Int{Value: 192},
&tengo.String{Value: "bar"})))
testBytecodeRemoveDuplicates(t,
bytecode(
concatInsts(
internal.MakeInstruction(internal.OpConstant, 0),
internal.MakeInstruction(internal.OpConstant, 1),
internal.MakeInstruction(internal.OpConstant, 2),
internal.MakeInstruction(internal.OpConstant, 3),
internal.MakeInstruction(internal.OpConstant, 4),
internal.MakeInstruction(internal.OpConstant, 5),
internal.MakeInstruction(internal.OpConstant, 6),
internal.MakeInstruction(internal.OpConstant, 7),
internal.MakeInstruction(internal.OpConstant, 8),
internal.MakeInstruction(internal.OpClosure, 4, 1)),
objectsArray(
&tengo.Int{Value: 1},
&tengo.Float{Value: 2.0},
&tengo.Char{Value: '3'},
&tengo.String{Value: "four"},
compiledFunction(1, 0,
internal.MakeInstruction(internal.OpConstant, 3),
internal.MakeInstruction(internal.OpConstant, 7),
internal.MakeInstruction(internal.OpSetLocal, 0),
internal.MakeInstruction(internal.OpGetGlobal, 0),
internal.MakeInstruction(internal.OpGetFree, 0)),
&tengo.Int{Value: 1},
&tengo.Float{Value: 2.0},
&tengo.Char{Value: '3'},
&tengo.String{Value: "four"})),
bytecode(
concatInsts(
internal.MakeInstruction(internal.OpConstant, 0),
internal.MakeInstruction(internal.OpConstant, 1),
internal.MakeInstruction(internal.OpConstant, 2),
internal.MakeInstruction(internal.OpConstant, 3),
internal.MakeInstruction(internal.OpConstant, 4),
internal.MakeInstruction(internal.OpConstant, 0),
internal.MakeInstruction(internal.OpConstant, 1),
internal.MakeInstruction(internal.OpConstant, 2),
internal.MakeInstruction(internal.OpConstant, 3),
internal.MakeInstruction(internal.OpClosure, 4, 1)),
objectsArray(
&tengo.Int{Value: 1},
&tengo.Float{Value: 2.0},
&tengo.Char{Value: '3'},
&tengo.String{Value: "four"},
compiledFunction(1, 0,
internal.MakeInstruction(internal.OpConstant, 3),
internal.MakeInstruction(internal.OpConstant, 2),
internal.MakeInstruction(internal.OpSetLocal, 0),
internal.MakeInstruction(internal.OpGetGlobal, 0),
internal.MakeInstruction(internal.OpGetFree, 0)))))
testBytecodeRemoveDuplicates(t,
bytecode(
concatInsts(
internal.MakeInstruction(internal.OpConstant, 0),
internal.MakeInstruction(internal.OpConstant, 1),
internal.MakeInstruction(internal.OpConstant, 2),
internal.MakeInstruction(internal.OpConstant, 3),
internal.MakeInstruction(internal.OpConstant, 4)),
objectsArray(
&tengo.Int{Value: 1},
&tengo.Int{Value: 2},
&tengo.Int{Value: 3},
&tengo.Int{Value: 1},
&tengo.Int{Value: 3})),
bytecode(
concatInsts(
internal.MakeInstruction(internal.OpConstant, 0),
internal.MakeInstruction(internal.OpConstant, 1),
internal.MakeInstruction(internal.OpConstant, 2),
internal.MakeInstruction(internal.OpConstant, 0),
internal.MakeInstruction(internal.OpConstant, 2)),
objectsArray(
&tengo.Int{Value: 1},
&tengo.Int{Value: 2},
&tengo.Int{Value: 3})))
}
func TestBytecode_CountObjects(t *testing.T) {
b := bytecode(
concatInsts(),
objectsArray(
&tengo.Int{Value: 55},
&tengo.Int{Value: 66},
&tengo.Int{Value: 77},
&tengo.Int{Value: 88},
compiledFunction(1, 0,
internal.MakeInstruction(internal.OpConstant, 3),
internal.MakeInstruction(internal.OpReturn, 1)),
compiledFunction(1, 0,
internal.MakeInstruction(internal.OpConstant, 2),
internal.MakeInstruction(internal.OpReturn, 1)),
compiledFunction(1, 0,
internal.MakeInstruction(internal.OpConstant, 1),
internal.MakeInstruction(internal.OpReturn, 1))))
require.Equal(t, 7, b.CountObjects())
}
func fileSet(files ...srcfile) *internal.SourceFileSet {
fileSet := internal.NewFileSet()
for _, f := range files {
fileSet.AddFile(f.name, -1, f.size)
}
return fileSet
}
func bytecodeFileSet(
instructions []byte,
constants []tengo.Object,
fileSet *internal.SourceFileSet,
) *tengo.Bytecode {
return &tengo.Bytecode{
FileSet: fileSet,
MainFunction: &tengo.CompiledFunction{Instructions: instructions},
Constants: constants,
}
}
func testBytecodeRemoveDuplicates(
t *testing.T,
input, expected *tengo.Bytecode,
) {
input.RemoveDuplicates()
require.Equal(t, expected.FileSet, input.FileSet)
require.Equal(t, expected.MainFunction, input.MainFunction)
require.Equal(t, expected.Constants, input.Constants)
}
func testBytecodeSerialization(t *testing.T, b *tengo.Bytecode) {
var buf bytes.Buffer
err := b.Encode(&buf)
require.NoError(t, err)
r := &tengo.Bytecode{}
err = r.Decode(bytes.NewReader(buf.Bytes()), nil)
require.NoError(t, err)
require.Equal(t, b.FileSet, r.FileSet)
require.Equal(t, b.MainFunction, r.MainFunction)
require.Equal(t, b.Constants, r.Constants)
}

329
cli/cli.go
View file

@ -1,329 +0,0 @@
package cli
import (
"bufio"
"bytes"
"fmt"
"io"
"io/ioutil"
"os"
"path/filepath"
"strings"
"github.com/d5/tengo/compiler"
"github.com/d5/tengo/compiler/ast"
"github.com/d5/tengo/compiler/parser"
"github.com/d5/tengo/compiler/source"
"github.com/d5/tengo/objects"
"github.com/d5/tengo/runtime"
)
const (
sourceFileExt = ".tengo"
replPrompt = ">> "
)
// Options represent CLI options
type Options struct {
// Compile output file
CompileOutput string
// Show help flag
ShowHelp bool
// Show version flag
ShowVersion bool
// Input file
InputFile string
// Version
Version string
// Import modules
Modules *objects.ModuleMap
}
// Run CLI
func Run(options *Options) {
if options.ShowHelp {
doHelp()
os.Exit(2)
} else if options.ShowVersion {
fmt.Println(options.Version)
return
}
if options.InputFile == "" {
// REPL
RunREPL(options.Modules, os.Stdin, os.Stdout)
return
}
inputData, err := ioutil.ReadFile(options.InputFile)
if err != nil {
_, _ = fmt.Fprintf(os.Stderr, "Error reading input file: %s", err.Error())
os.Exit(1)
}
if options.CompileOutput != "" {
if err := CompileOnly(options.Modules, inputData, options.InputFile, options.CompileOutput); err != nil {
_, _ = fmt.Fprintln(os.Stderr, err.Error())
os.Exit(1)
}
} else if filepath.Ext(options.InputFile) == sourceFileExt {
if err := CompileAndRun(options.Modules, inputData, options.InputFile); err != nil {
_, _ = fmt.Fprintln(os.Stderr, err.Error())
os.Exit(1)
}
} else {
if err := RunCompiled(options.Modules, inputData); err != nil {
_, _ = fmt.Fprintln(os.Stderr, err.Error())
os.Exit(1)
}
}
}
func doHelp() {
fmt.Println("Usage:")
fmt.Println()
fmt.Println(" tengo [flags] {input-file}")
fmt.Println()
fmt.Println("Flags:")
fmt.Println()
fmt.Println(" -o compile output file")
fmt.Println(" -version show version")
fmt.Println()
fmt.Println("Examples:")
fmt.Println()
fmt.Println(" tengo")
fmt.Println()
fmt.Println(" Start Tengo REPL")
fmt.Println()
fmt.Println(" tengo myapp.tengo")
fmt.Println()
fmt.Println(" Compile and run source file (myapp.tengo)")
fmt.Println(" Source file must have .tengo extension")
fmt.Println()
fmt.Println(" tengo -o myapp myapp.tengo")
fmt.Println()
fmt.Println(" Compile source file (myapp.tengo) into bytecode file (myapp)")
fmt.Println()
fmt.Println(" tengo myapp")
fmt.Println()
fmt.Println(" Run bytecode file (myapp)")
fmt.Println()
fmt.Println()
}
// CompileOnly compiles the source code and writes the compiled binary into outputFile.
func CompileOnly(modules *objects.ModuleMap, data []byte, inputFile, outputFile string) (err error) {
bytecode, err := compileSrc(modules, data, filepath.Base(inputFile))
if err != nil {
return
}
if outputFile == "" {
outputFile = basename(inputFile) + ".out"
}
out, err := os.OpenFile(outputFile, os.O_CREATE|os.O_WRONLY, os.ModePerm)
if err != nil {
return
}
defer func() {
if err != nil {
_ = out.Close()
} else {
err = out.Close()
}
}()
err = bytecode.Encode(out)
if err != nil {
return
}
fmt.Println(outputFile)
return
}
// CompileAndRun compiles the source code and executes it.
func CompileAndRun(modules *objects.ModuleMap, data []byte, inputFile string) (err error) {
bytecode, err := compileSrc(modules, data, filepath.Base(inputFile))
if err != nil {
return
}
machine := runtime.NewVM(bytecode, nil, -1)
err = machine.Run()
if err != nil {
return
}
return
}
// RunCompiled reads the compiled binary from file and executes it.
func RunCompiled(modules *objects.ModuleMap, data []byte) (err error) {
bytecode := &compiler.Bytecode{}
err = bytecode.Decode(bytes.NewReader(data), modules)
if err != nil {
return
}
machine := runtime.NewVM(bytecode, nil, -1)
err = machine.Run()
if err != nil {
return
}
return
}
// RunREPL starts REPL.
func RunREPL(modules *objects.ModuleMap, in io.Reader, out io.Writer) {
stdin := bufio.NewScanner(in)
fileSet := source.NewFileSet()
globals := make([]objects.Object, runtime.GlobalsSize)
symbolTable := compiler.NewSymbolTable()
for idx, fn := range objects.Builtins {
symbolTable.DefineBuiltin(idx, fn.Name)
}
// embed println function
symbol := symbolTable.Define("__repl_println__")
globals[symbol.Index] = &objects.UserFunction{
Name: "println",
Value: func(args ...objects.Object) (ret objects.Object, err error) {
var printArgs []interface{}
for _, arg := range args {
if _, isUndefined := arg.(*objects.Undefined); isUndefined {
printArgs = append(printArgs, "<undefined>")
} else {
s, _ := objects.ToString(arg)
printArgs = append(printArgs, s)
}
}
printArgs = append(printArgs, "\n")
_, _ = fmt.Print(printArgs...)
return
},
}
var constants []objects.Object
for {
_, _ = fmt.Fprint(out, replPrompt)
scanned := stdin.Scan()
if !scanned {
return
}
line := stdin.Text()
srcFile := fileSet.AddFile("repl", -1, len(line))
p := parser.NewParser(srcFile, []byte(line), nil)
file, err := p.ParseFile()
if err != nil {
_, _ = fmt.Fprintln(out, err.Error())
continue
}
file = addPrints(file)
c := compiler.NewCompiler(srcFile, symbolTable, constants, modules, nil)
if err := c.Compile(file); err != nil {
_, _ = fmt.Fprintln(out, err.Error())
continue
}
bytecode := c.Bytecode()
machine := runtime.NewVM(bytecode, globals, -1)
if err := machine.Run(); err != nil {
_, _ = fmt.Fprintln(out, err.Error())
continue
}
constants = bytecode.Constants
}
}
func compileSrc(modules *objects.ModuleMap, src []byte, filename string) (*compiler.Bytecode, error) {
fileSet := source.NewFileSet()
srcFile := fileSet.AddFile(filename, -1, len(src))
p := parser.NewParser(srcFile, src, nil)
file, err := p.ParseFile()
if err != nil {
return nil, err
}
c := compiler.NewCompiler(srcFile, nil, nil, modules, nil)
c.EnableFileImport(true)
if err := c.Compile(file); err != nil {
return nil, err
}
bytecode := c.Bytecode()
bytecode.RemoveDuplicates()
return bytecode, nil
}
func addPrints(file *ast.File) *ast.File {
var stmts []ast.Stmt
for _, s := range file.Stmts {
switch s := s.(type) {
case *ast.ExprStmt:
stmts = append(stmts, &ast.ExprStmt{
Expr: &ast.CallExpr{
Func: &ast.Ident{Name: "__repl_println__"},
Args: []ast.Expr{s.Expr},
},
})
case *ast.AssignStmt:
stmts = append(stmts, s)
stmts = append(stmts, &ast.ExprStmt{
Expr: &ast.CallExpr{
Func: &ast.Ident{
Name: "__repl_println__",
},
Args: s.LHS,
},
})
default:
stmts = append(stmts, s)
}
}
return &ast.File{
InputFile: file.InputFile,
Stmts: stmts,
}
}
func basename(s string) string {
s = filepath.Base(s)
n := strings.LastIndexByte(s, '.')
if n > 0 {
return s[:n]
}
return s
}

64
cli/cli_test.go
View file

@ -1,64 +0,0 @@
package cli_test
import (
"io/ioutil"
"os"
"path/filepath"
"regexp"
"testing"
"github.com/d5/tengo/assert"
"github.com/d5/tengo/cli"
"github.com/d5/tengo/stdlib"
)
func TestCLICompileAndRun(t *testing.T) {
tempDir := filepath.Join(os.TempDir(), "tengo_tests")
_ = os.MkdirAll(tempDir, os.ModePerm)
binFile := filepath.Join(tempDir, "cli_bin")
outFile := filepath.Join(tempDir, "cli_out")
defer func() {
_ = os.RemoveAll(tempDir)
}()
src := []byte(`
os := import("os")
rand := import("rand")
times := import("times")
rand.seed(times.time_nanosecond(times.now()))
rand_num := func() {
return rand.intn(100)
}
file := os.create("` + outFile + `")
file.write_string("random number is " + rand_num())
file.close()
`)
mods := stdlib.GetModuleMap(stdlib.AllModuleNames()...)
err := cli.CompileOnly(mods, src, "src", binFile)
if !assert.NoError(t, err) {
return
}
compiledBin, err := ioutil.ReadFile(binFile)
if !assert.NoError(t, err) {
return
}
err = cli.RunCompiled(mods, compiledBin)
if !assert.NoError(t, err) {
return
}
read, err := ioutil.ReadFile(outFile)
if !assert.NoError(t, err) {
return
}
ok, err := regexp.Match(`^random number is \d+$`, read)
assert.NoError(t, err)
assert.True(t, ok, string(read))
}

57
cmd/bench/main.go
View file

@ -4,12 +4,8 @@ import (
"fmt"
"time"
"github.com/d5/tengo/compiler"
"github.com/d5/tengo/compiler/ast"
"github.com/d5/tengo/compiler/parser"
"github.com/d5/tengo/compiler/source"
"github.com/d5/tengo/objects"
"github.com/d5/tengo/runtime"
"github.com/d5/tengo"
"github.com/d5/tengo/internal"
)
func main() {
@ -40,8 +36,9 @@ fib := func(x) {
panic(err)
}
if nativeResult != int(result.(*objects.Int).Value) {
panic(fmt.Errorf("wrong result: %d != %d", nativeResult, int(result.(*objects.Int).Value)))
if nativeResult != int(result.(*tengo.Int).Value) {
panic(fmt.Errorf("wrong result: %d != %d", nativeResult,
int(result.(*tengo.Int).Value)))
}
fmt.Println("-------------------------------------")
@ -76,8 +73,9 @@ fib := func(x, s) {
panic(err)
}
if nativeResult != int(result.(*objects.Int).Value) {
panic(fmt.Errorf("wrong result: %d != %d", nativeResult, int(result.(*objects.Int).Value)))
if nativeResult != int(result.(*tengo.Int).Value) {
panic(fmt.Errorf("wrong result: %d != %d", nativeResult,
int(result.(*tengo.Int).Value)))
}
fmt.Println("-------------------------------------")
@ -112,8 +110,9 @@ fib := func(x, a, b) {
panic(err)
}
if nativeResult != int(result.(*objects.Int).Value) {
panic(fmt.Errorf("wrong result: %d != %d", nativeResult, int(result.(*objects.Int).Value)))
if nativeResult != int(result.(*tengo.Int).Value) {
panic(fmt.Errorf("wrong result: %d != %d", nativeResult,
int(result.(*tengo.Int).Value)))
}
fmt.Println("-------------------------------------")
@ -155,14 +154,22 @@ func fibTC2(n, a, b int) int {
}
}
func runBench(input []byte) (parseTime time.Duration, compileTime time.Duration, runTime time.Duration, result objects.Object, err error) {
var astFile *ast.File
func runBench(
input []byte,
) (
parseTime time.Duration,
compileTime time.Duration,
runTime time.Duration,
result tengo.Object,
err error,
) {
var astFile *internal.File
parseTime, astFile, err = parse(input)
if err != nil {
return
}
var bytecode *compiler.Bytecode
var bytecode *tengo.Bytecode
compileTime, bytecode, err = compileFile(astFile)
if err != nil {
return
@ -173,13 +180,13 @@ func runBench(input []byte) (parseTime time.Duration, compileTime time.Duration,
return
}
func parse(input []byte) (time.Duration, *ast.File, error) {
fileSet := source.NewFileSet()
func parse(input []byte) (time.Duration, *internal.File, error) {
fileSet := internal.NewFileSet()
inputFile := fileSet.AddFile("bench", -1, len(input))
start := time.Now()
p := parser.NewParser(inputFile, input, nil)
p := internal.NewParser(inputFile, input, nil)
file, err := p.ParseFile()
if err != nil {
return time.Since(start), nil, err
@ -188,13 +195,13 @@ func parse(input []byte) (time.Duration, *ast.File, error) {
return time.Since(start), file, nil
}
func compileFile(file *ast.File) (time.Duration, *compiler.Bytecode, error) {
symTable := compiler.NewSymbolTable()
func compileFile(file *internal.File) (time.Duration, *tengo.Bytecode, error) {
symTable := internal.NewSymbolTable()
symTable.Define("out")
start := time.Now()
c := compiler.NewCompiler(file.InputFile, symTable, nil, nil, nil)
c := tengo.NewCompiler(file.InputFile, symTable, nil, nil, nil)
if err := c.Compile(file); err != nil {
return time.Since(start), nil, err
}
@ -205,12 +212,14 @@ func compileFile(file *ast.File) (time.Duration, *compiler.Bytecode, error) {
return time.Since(start), bytecode, nil
}
func runVM(bytecode *compiler.Bytecode) (time.Duration, objects.Object, error) {
globals := make([]objects.Object, runtime.GlobalsSize)
func runVM(
bytecode *tengo.Bytecode,
) (time.Duration, tengo.Object, error) {
globals := make([]tengo.Object, tengo.GlobalsSize)
start := time.Now()
v := runtime.NewVM(bytecode, globals, -1)
v := tengo.NewVM(bytecode, globals, -1)
if err := v.Run(); err != nil {
return time.Since(start), nil, err
}

294
cmd/tengo/main.go
View file

@ -1,12 +1,26 @@
package main
import (
"bufio"
"bytes"
"flag"
"fmt"
"io"
"io/ioutil"
"os"
"path/filepath"
"strings"
"github.com/d5/tengo/cli"
"github.com/d5/tengo"
"github.com/d5/tengo/internal"
"github.com/d5/tengo/stdlib"
)
const (
sourceFileExt = ".tengo"
replPrompt = ">> "
)
var (
compileOutput string
showHelp bool
@ -22,12 +36,274 @@ func init() {
}
func main() {
cli.Run(&cli.Options{
ShowHelp: showHelp,
ShowVersion: showVersion,
Version: version,
CompileOutput: compileOutput,
Modules: stdlib.GetModuleMap(stdlib.AllModuleNames()...),
InputFile: flag.Arg(0),
})
if showHelp {
doHelp()
os.Exit(2)
} else if showVersion {
fmt.Println(version)
return
}
modules := stdlib.GetModuleMap(stdlib.AllModuleNames()...)
inputFile := flag.Arg(0)
if inputFile == "" {
// REPL
RunREPL(modules, os.Stdin, os.Stdout)
return
}
inputData, err := ioutil.ReadFile(inputFile)
if err != nil {
_, _ = fmt.Fprintf(os.Stderr,
"Error reading input file: %s", err.Error())
os.Exit(1)
}
if compileOutput != "" {
err := CompileOnly(modules, inputData, inputFile,
compileOutput)
if err != nil {
_, _ = fmt.Fprintln(os.Stderr, err.Error())
os.Exit(1)
}
} else if filepath.Ext(inputFile) == sourceFileExt {
err := CompileAndRun(modules, inputData, inputFile)
if err != nil {
_, _ = fmt.Fprintln(os.Stderr, err.Error())
os.Exit(1)
}
} else {
if err := RunCompiled(modules, inputData); err != nil {
_, _ = fmt.Fprintln(os.Stderr, err.Error())
os.Exit(1)
}
}
}
// CompileOnly compiles the source code and writes the compiled binary into
// outputFile.
func CompileOnly(
modules *tengo.ModuleMap,
data []byte,
inputFile, outputFile string,
) (err error) {
bytecode, err := compileSrc(modules, data, filepath.Base(inputFile))
if err != nil {
return
}
if outputFile == "" {
outputFile = basename(inputFile) + ".out"
}
out, err := os.OpenFile(outputFile, os.O_CREATE|os.O_WRONLY, os.ModePerm)
if err != nil {
return
}
defer func() {
if err != nil {
_ = out.Close()
} else {
err = out.Close()
}
}()
err = bytecode.Encode(out)
if err != nil {
return
}
fmt.Println(outputFile)
return
}
// CompileAndRun compiles the source code and executes it.
func CompileAndRun(
modules *tengo.ModuleMap,
data []byte,
inputFile string,
) (err error) {
bytecode, err := compileSrc(modules, data, filepath.Base(inputFile))
if err != nil {
return
}
machine := tengo.NewVM(bytecode, nil, -1)
err = machine.Run()
return
}
// RunCompiled reads the compiled binary from file and executes it.
func RunCompiled(modules *tengo.ModuleMap, data []byte) (err error) {
bytecode := &tengo.Bytecode{}
err = bytecode.Decode(bytes.NewReader(data), modules)
if err != nil {
return
}
machine := tengo.NewVM(bytecode, nil, -1)
err = machine.Run()
return
}
// RunREPL starts REPL.
func RunREPL(modules *tengo.ModuleMap, in io.Reader, out io.Writer) {
stdin := bufio.NewScanner(in)
fileSet := internal.NewFileSet()
globals := make([]tengo.Object, tengo.GlobalsSize)
symbolTable := internal.NewSymbolTable()
for idx, fn := range tengo.GetAllBuiltinFunctions() {
symbolTable.DefineBuiltin(idx, fn.Name)
}
// embed println function
symbol := symbolTable.Define("__repl_println__")
globals[symbol.Index] = &tengo.UserFunction{
Name: "println",
Value: func(args ...tengo.Object) (ret tengo.Object, err error) {
var printArgs []interface{}
for _, arg := range args {
if _, isUndefined := arg.(*tengo.Undefined); isUndefined {
printArgs = append(printArgs, "<undefined>")
} else {
s, _ := tengo.ToString(arg)
printArgs = append(printArgs, s)
}
}
printArgs = append(printArgs, "\n")
_, _ = fmt.Print(printArgs...)
return
},
}
var constants []tengo.Object
for {
_, _ = fmt.Fprint(out, replPrompt)
scanned := stdin.Scan()
if !scanned {
return
}
line := stdin.Text()
srcFile := fileSet.AddFile("repl", -1, len(line))
p := internal.NewParser(srcFile, []byte(line), nil)
file, err := p.ParseFile()
if err != nil {
_, _ = fmt.Fprintln(out, err.Error())
continue
}
file = addPrints(file)
c := tengo.NewCompiler(srcFile, symbolTable, constants, modules, nil)
if err := c.Compile(file); err != nil {
_, _ = fmt.Fprintln(out, err.Error())
continue
}
bytecode := c.Bytecode()
machine := tengo.NewVM(bytecode, globals, -1)
if err := machine.Run(); err != nil {
_, _ = fmt.Fprintln(out, err.Error())
continue
}
constants = bytecode.Constants
}
}
func compileSrc(
modules *tengo.ModuleMap,
src []byte,
filename string,
) (*tengo.Bytecode, error) {
fileSet := internal.NewFileSet()
srcFile := fileSet.AddFile(filename, -1, len(src))
p := internal.NewParser(srcFile, src, nil)
file, err := p.ParseFile()
if err != nil {
return nil, err
}
c := tengo.NewCompiler(srcFile, nil, nil, modules, nil)
c.EnableFileImport(true)
if err := c.Compile(file); err != nil {
return nil, err
}
bytecode := c.Bytecode()
bytecode.RemoveDuplicates()
return bytecode, nil
}
func doHelp() {
fmt.Println("Usage:")
fmt.Println()
fmt.Println(" tengo [flags] {input-file}")
fmt.Println()
fmt.Println("Flags:")
fmt.Println()
fmt.Println(" -o compile output file")
fmt.Println(" -version show version")
fmt.Println()
fmt.Println("Examples:")
fmt.Println()
fmt.Println(" tengo")
fmt.Println()
fmt.Println(" Start Tengo REPL")
fmt.Println()
fmt.Println(" tengo myapp.tengo")
fmt.Println()
fmt.Println(" Compile and run source file (myapp.tengo)")
fmt.Println(" Source file must have .tengo extension")
fmt.Println()
fmt.Println(" tengo -o myapp myapp.tengo")
fmt.Println()
fmt.Println(" Compile source file (myapp.tengo) into bytecode file (myapp)")
fmt.Println()
fmt.Println(" tengo myapp")
fmt.Println()
fmt.Println(" Run bytecode file (myapp)")
fmt.Println()
fmt.Println()
}
func addPrints(file *internal.File) *internal.File {
var stmts []internal.Stmt
for _, s := range file.Stmts {
switch s := s.(type) {
case *internal.ExprStmt:
stmts = append(stmts, &internal.ExprStmt{
Expr: &internal.CallExpr{
Func: &internal.Ident{Name: "__repl_println__"},
Args: []internal.Expr{s.Expr},
},
})
case *internal.AssignStmt:
stmts = append(stmts, s)
stmts = append(stmts, &internal.ExprStmt{
Expr: &internal.CallExpr{
Func: &internal.Ident{
Name: "__repl_println__",
},
Args: s.LHS,
},
})
default:
stmts = append(stmts, s)
}
}
return &internal.File{
InputFile: file.InputFile,
Stmts: stmts,
}
}
func basename(s string) string {
s = filepath.Base(s)
n := strings.LastIndexByte(s, '.')
if n > 0 {
return s[:n]
}
return s
}

31
cmd/tengomin/main.go
View file

@ -1,31 +0,0 @@
package main
import (
"flag"
"github.com/d5/tengo/cli"
)
var (
compileOutput string
showHelp bool
showVersion bool
version = "dev"
)
func init() {
flag.BoolVar(&showHelp, "help", false, "Show help")
flag.StringVar(&compileOutput, "o", "", "Compile output file")
flag.BoolVar(&showVersion, "version", false, "Show version")
flag.Parse()
}
func main() {
cli.Run(&cli.Options{
ShowHelp: showHelp,
ShowVersion: showVersion,
Version: version,
CompileOutput: compileOutput,
InputFile: flag.Arg(0),
})
}

1285
compiler.go Normal file
View file

File diff suppressed because it is too large Load diff

35
compiler/ast/array_lit.go
View file

@ -1,35 +0,0 @@
package ast
import (
"strings"
"github.com/d5/tengo/compiler/source"
)
// ArrayLit represents an array literal.
type ArrayLit struct {
Elements []Expr
LBrack source.Pos
RBrack source.Pos
}
func (e *ArrayLit) exprNode() {}
// Pos returns the position of first character belonging to the node.
func (e *ArrayLit) Pos() source.Pos {
return e.LBrack
}
// End returns the position of first character immediately after the node.
func (e *ArrayLit) End() source.Pos {
return e.RBrack + 1
}
func (e *ArrayLit) String() string {
var elements []string
for _, m := range e.Elements {
elements = append(elements, m.String())
}
return "[" + strings.Join(elements, ", ") + "]"
}

40
compiler/ast/assign_stmt.go
View file

@ -1,40 +0,0 @@
package ast
import (
"strings"
"github.com/d5/tengo/compiler/source"
"github.com/d5/tengo/compiler/token"
)
// AssignStmt represents an assignment statement.
type AssignStmt struct {
LHS []Expr
RHS []Expr
Token token.Token
TokenPos source.Pos
}
func (s *AssignStmt) stmtNode() {}
// Pos returns the position of first character belonging to the node.
func (s *AssignStmt) Pos() source.Pos {
return s.LHS[0].Pos()
}
// End returns the position of first character immediately after the node.
func (s *AssignStmt) End() source.Pos {
return s.RHS[len(s.RHS)-1].End()
}
func (s *AssignStmt) String() string {
var lhs, rhs []string
for _, e := range s.LHS {
lhs = append(lhs, e.String())
}
for _, e := range s.RHS {
rhs = append(rhs, e.String())
}
return strings.Join(lhs, ", ") + " " + s.Token.String() + " " + strings.Join(rhs, ", ")
}

5
compiler/ast/ast.go
View file

@ -1,5 +0,0 @@
package ast
const (
nullRep = "<null>"
)

25
compiler/ast/bad_expr.go
View file

@ -1,25 +0,0 @@
package ast
import "github.com/d5/tengo/compiler/source"
// BadExpr represents a bad expression.
type BadExpr struct {
From source.Pos
To source.Pos
}
func (e *BadExpr) exprNode() {}
// Pos returns the position of first character belonging to the node.
func (e *BadExpr) Pos() source.Pos {
return e.From
}
// End returns the position of first character immediately after the node.
func (e *BadExpr) End() source.Pos {
return e.To
}
func (e *BadExpr) String() string {
return "<bad expression>"
}

25
compiler/ast/bad_stmt.go
View file

@ -1,25 +0,0 @@
package ast
import "github.com/d5/tengo/compiler/source"
// BadStmt represents a bad statement.
type BadStmt struct {
From source.Pos
To source.Pos
}
func (s *BadStmt) stmtNode() {}
// Pos returns the position of first character belonging to the node.
func (s *BadStmt) Pos() source.Pos {
return s.From
}
// End returns the position of first character immediately after the node.
func (s *BadStmt) End() source.Pos {
return s.To
}
func (s *BadStmt) String() string {
return "<bad statement>"
}

30
compiler/ast/binary_expr.go
View file

@ -1,30 +0,0 @@
package ast
import (
"github.com/d5/tengo/compiler/source"
"github.com/d5/tengo/compiler/token"
)
// BinaryExpr represents a binary operator expression.
type BinaryExpr struct {
LHS Expr
RHS Expr
Token token.Token
TokenPos source.Pos
}
func (e *BinaryExpr) exprNode() {}
// Pos returns the position of first character belonging to the node.
func (e *BinaryExpr) Pos() source.Pos {
return e.LHS.Pos()
}
// End returns the position of first character immediately after the node.
func (e *BinaryExpr) End() source.Pos {
return e.RHS.End()
}
func (e *BinaryExpr) String() string {
return "(" + e.LHS.String() + " " + e.Token.String() + " " + e.RHS.String() + ")"
}

35
compiler/ast/block_stmt.go
View file

@ -1,35 +0,0 @@
package ast
import (
"strings"
"github.com/d5/tengo/compiler/source"
)
// BlockStmt represents a block statement.
type BlockStmt struct {
Stmts []Stmt
LBrace source.Pos
RBrace source.Pos
}
func (s *BlockStmt) stmtNode() {}
// Pos returns the position of first character belonging to the node.
func (s *BlockStmt) Pos() source.Pos {
return s.LBrace
}
// End returns the position of first character immediately after the node.
func (s *BlockStmt) End() source.Pos {
return s.RBrace + 1
}
func (s *BlockStmt) String() string {
var list []string
for _, e := range s.Stmts {
list = append(list, e.String())
}
return "{" + strings.Join(list, "; ") + "}"
}

26
compiler/ast/bool_lit.go
View file

@ -1,26 +0,0 @@
package ast
import "github.com/d5/tengo/compiler/source"
// BoolLit represents a boolean literal.
type BoolLit struct {
Value bool
ValuePos source.Pos
Literal string
}
func (e *BoolLit) exprNode() {}
// Pos returns the position of first character belonging to the node.
func (e *BoolLit) Pos() source.Pos {
return e.ValuePos
}
// End returns the position of first character immediately after the node.
func (e *BoolLit) End() source.Pos {
return source.Pos(int(e.ValuePos) + len(e.Literal))
}
func (e *BoolLit) String() string {
return e.Literal
}

38
compiler/ast/branch_stmt.go
View file

@ -1,38 +0,0 @@
package ast
import (
"github.com/d5/tengo/compiler/source"
"github.com/d5/tengo/compiler/token"
)
// BranchStmt represents a branch statement.
type BranchStmt struct {
Token token.Token
TokenPos source.Pos
Label *Ident
}
func (s *BranchStmt) stmtNode() {}
// Pos returns the position of first character belonging to the node.
func (s *BranchStmt) Pos() source.Pos {
return s.TokenPos
}
// End returns the position of first character immediately after the node.
func (s *BranchStmt) End() source.Pos {
if s.Label != nil {
return s.Label.End()
}
return source.Pos(int(s.TokenPos) + len(s.Token.String()))
}
func (s *BranchStmt) String() string {
var label string
if s.Label != nil {
label = " " + s.Label.Name
}
return s.Token.String() + label
}

36
compiler/ast/call_expr.go
View file

@ -1,36 +0,0 @@
package ast
import (
"strings"
"github.com/d5/tengo/compiler/source"
)
// CallExpr represents a function call expression.
type CallExpr struct {
Func Expr
LParen source.Pos
Args []Expr
RParen source.Pos
}
func (e *CallExpr) exprNode() {}
// Pos returns the position of first character belonging to the node.
func (e *CallExpr) Pos() source.Pos {
return e.Func.Pos()
}
// End returns the position of first character immediately after the node.
func (e *CallExpr) End() source.Pos {
return e.RParen + 1
}
func (e *CallExpr) String() string {
var args []string
for _, e := range e.Args {
args = append(args, e.String())
}
return e.Func.String() + "(" + strings.Join(args, ", ") + ")"
}

26
compiler/ast/char_lit.go
View file

@ -1,26 +0,0 @@
package ast
import "github.com/d5/tengo/compiler/source"
// CharLit represents a character literal.
type CharLit struct {
Value rune
ValuePos source.Pos
Literal string
}
func (e *CharLit) exprNode() {}
// Pos returns the position of first character belonging to the node.
func (e *CharLit) Pos() source.Pos {
return e.ValuePos
}
// End returns the position of first character immediately after the node.
func (e *CharLit) End() source.Pos {
return source.Pos(int(e.ValuePos) + len(e.Literal))
}
func (e *CharLit) String() string {
return e.Literal
}

30
compiler/ast/cond_expr.go
View file

@ -1,30 +0,0 @@
package ast
import (
"github.com/d5/tengo/compiler/source"
)
// CondExpr represents a ternary conditional expression.
type CondExpr struct {
Cond Expr
True Expr
False Expr
QuestionPos source.Pos
ColonPos source.Pos
}
func (e *CondExpr) exprNode() {}
// Pos returns the position of first character belonging to the node.
func (e *CondExpr) Pos() source.Pos {
return e.Cond.Pos()
}
// End returns the position of first character immediately after the node.
func (e *CondExpr) End() source.Pos {
return e.False.End()
}
func (e *CondExpr) String() string {
return "(" + e.Cond.String() + " ? " + e.True.String() + " : " + e.False.String() + ")"
}

29
compiler/ast/empty_stmt.go
View file

@ -1,29 +0,0 @@
package ast
import "github.com/d5/tengo/compiler/source"
// EmptyStmt represents an empty statement.
type EmptyStmt struct {
Semicolon source.Pos
Implicit bool
}
func (s *EmptyStmt) stmtNode() {}
// Pos returns the position of first character belonging to the node.
func (s *EmptyStmt) Pos() source.Pos {
return s.Semicolon
}
// End returns the position of first character immediately after the node.
func (s *EmptyStmt) End() source.Pos {
if s.Implicit {
return s.Semicolon
}
return s.Semicolon + 1
}
func (s *EmptyStmt) String() string {
return ";"
}

29
compiler/ast/error_expr.go
View file

@ -1,29 +0,0 @@
package ast
import (
"github.com/d5/tengo/compiler/source"
)
// ErrorExpr represents an error expression
type ErrorExpr struct {
Expr Expr
ErrorPos source.Pos
LParen source.Pos
RParen source.Pos
}
func (e *ErrorExpr) exprNode() {}
// Pos returns the position of first character belonging to the node.
func (e *ErrorExpr) Pos() source.Pos {
return e.ErrorPos
}
// End returns the position of first character immediately after the node.
func (e *ErrorExpr) End() source.Pos {
return e.RParen
}
func (e *ErrorExpr) String() string {
return "error(" + e.Expr.String() + ")"
}

27
compiler/ast/export_stmt.go
View file

@ -1,27 +0,0 @@
package ast
import (
"github.com/d5/tengo/compiler/source"
)
// ExportStmt represents an export statement.
type ExportStmt struct {
ExportPos source.Pos
Result Expr
}
func (s *ExportStmt) stmtNode() {}
// Pos returns the position of first character belonging to the node.
func (s *ExportStmt) Pos() source.Pos {
return s.ExportPos
}
// End returns the position of first character immediately after the node.
func (s *ExportStmt) End() source.Pos {
return s.Result.End()
}
func (s *ExportStmt) String() string {
return "export " + s.Result.String()
}

7
compiler/ast/expr.go
View file

@ -1,7 +0,0 @@
package ast
// Expr represents an expression node in the AST.
type Expr interface {
Node
exprNode()
}

24
compiler/ast/expr_stmt.go
View file

@ -1,24 +0,0 @@
package ast
import "github.com/d5/tengo/compiler/source"
// ExprStmt represents an expression statement.
type ExprStmt struct {
Expr Expr
}
func (s *ExprStmt) stmtNode() {}
// Pos returns the position of first character belonging to the node.
func (s *ExprStmt) Pos() source.Pos {
return s.Expr.Pos()
}
// End returns the position of first character immediately after the node.
func (s *ExprStmt) End() source.Pos {
return s.Expr.End()
}
func (s *ExprStmt) String() string {
return s.Expr.String()
}

26
compiler/ast/float_lit.go
View file

@ -1,26 +0,0 @@
package ast
import "github.com/d5/tengo/compiler/source"
// FloatLit represents a floating point literal.
type FloatLit struct {
Value float64
ValuePos source.Pos
Literal string
}
func (e *FloatLit) exprNode() {}
// Pos returns the position of first character belonging to the node.
func (e *FloatLit) Pos() source.Pos {
return e.ValuePos
}
// End returns the position of first character immediately after the node.
func (e *FloatLit) End() source.Pos {
return source.Pos(int(e.ValuePos) + len(e.Literal))
}
func (e *FloatLit) String() string {
return e.Literal
}

32
compiler/ast/for_in_stmt.go
View file

@ -1,32 +0,0 @@
package ast
import "github.com/d5/tengo/compiler/source"
// ForInStmt represents a for-in statement.
type ForInStmt struct {
ForPos source.Pos
Key *Ident
Value *Ident
Iterable Expr
Body *BlockStmt
}
func (s *ForInStmt) stmtNode() {}
// Pos returns the position of first character belonging to the node.
func (s *ForInStmt) Pos() source.Pos {
return s.ForPos
}
// End returns the position of first character immediately after the node.
func (s *ForInStmt) End() source.Pos {
return s.Body.End()
}
func (s *ForInStmt) String() string {
if s.Value != nil {
return "for " + s.Key.String() + ", " + s.Value.String() + " in " + s.Iterable.String() + " " + s.Body.String()
}
return "for " + s.Key.String() + " in " + s.Iterable.String() + " " + s.Body.String()
}

43
compiler/ast/for_stmt.go
View file

@ -1,43 +0,0 @@
package ast
import "github.com/d5/tengo/compiler/source"
// ForStmt represents a for statement.
type ForStmt struct {
ForPos source.Pos
Init Stmt
Cond Expr
Post Stmt
Body *BlockStmt
}
func (s *ForStmt) stmtNode() {}
// Pos returns the position of first character belonging to the node.
func (s *ForStmt) Pos() source.Pos {
return s.ForPos
}
// End returns the position of first character immediately after the node.
func (s *ForStmt) End() source.Pos {
return s.Body.End()
}
func (s *ForStmt) String() string {
var init, cond, post string
if s.Init != nil {
init = s.Init.String()
}
if s.Cond != nil {
cond = s.Cond.String() + " "
}
if s.Post != nil {
post = s.Post.String()
}
if init != "" || post != "" {
return "for " + init + " ; " + cond + " ; " + post + s.Body.String()
}
return "for " + cond + s.Body.String()
}

25
compiler/ast/func_lit.go
View file

@ -1,25 +0,0 @@
package ast
import "github.com/d5/tengo/compiler/source"
// FuncLit represents a function literal.
type FuncLit struct {
Type *FuncType
Body *BlockStmt
}
func (e *FuncLit) exprNode() {}
// Pos returns the position of first character belonging to the node.
func (e *FuncLit) Pos() source.Pos {
return e.Type.Pos()
}
// End returns the position of first character immediately after the node.
func (e *FuncLit) End() source.Pos {
return e.Body.End()
}
func (e *FuncLit) String() string {
return "func" + e.Type.Params.String() + " " + e.Body.String()
}

25
compiler/ast/func_type.go
View file

@ -1,25 +0,0 @@
package ast
import "github.com/d5/tengo/compiler/source"
// FuncType represents a function type definition.
type FuncType struct {
FuncPos source.Pos
Params *IdentList
}
func (e *FuncType) exprNode() {}
// Pos returns the position of first character belonging to the node.
func (e *FuncType) Pos() source.Pos {
return e.FuncPos
}
// End returns the position of first character immediately after the node.
func (e *FuncType) End() source.Pos {
return e.Params.End()
}
func (e *FuncType) String() string {
return "func" + e.Params.String()
}

29
compiler/ast/ident.go
View file

@ -1,29 +0,0 @@
package ast
import "github.com/d5/tengo/compiler/source"
// Ident represents an identifier.
type Ident struct {
Name string
NamePos source.Pos
}
func (e *Ident) exprNode() {}
// Pos returns the position of first character belonging to the node.
func (e *Ident) Pos() source.Pos {
return e.NamePos
}
// End returns the position of first character immediately after the node.
func (e *Ident) End() source.Pos {
return source.Pos(int(e.NamePos) + len(e.Name))
}
func (e *Ident) String() string {
if e != nil {
return e.Name
}
return nullRep
}

40
compiler/ast/if_stmt.go
View file

@ -1,40 +0,0 @@
package ast
import "github.com/d5/tengo/compiler/source"
// IfStmt represents an if statement.
type IfStmt struct {
IfPos source.Pos
Init Stmt
Cond Expr
Body *BlockStmt
Else Stmt // else branch; or nil
}
func (s *IfStmt) stmtNode() {}
// Pos returns the position of first character belonging to the node.
func (s *IfStmt) Pos() source.Pos {
return s.IfPos
}
// End returns the position of first character immediately after the node.
func (s *IfStmt) End() source.Pos {
if s.Else != nil {
return s.Else.End()
}
return s.Body.End()
}
func (s *IfStmt) String() string {
var initStmt, elseStmt string
if s.Init != nil {
initStmt = s.Init.String() + "; "
}
if s.Else != nil {
elseStmt = " else " + s.Else.String()
}
return "if " + initStmt + s.Cond.String() + " " + s.Body.String() + elseStmt
}

29
compiler/ast/immutable_expr.go
View file

@ -1,29 +0,0 @@
package ast
import (
"github.com/d5/tengo/compiler/source"
)
// ImmutableExpr represents an immutable expression
type ImmutableExpr struct {
Expr Expr
ErrorPos source.Pos
LParen source.Pos
RParen source.Pos
}
func (e *ImmutableExpr) exprNode() {}
// Pos returns the position of first character belonging to the node.
func (e *ImmutableExpr) Pos() source.Pos {
return e.ErrorPos
}
// End returns the position of first character immediately after the node.
func (e *ImmutableExpr) End() source.Pos {
return e.RParen
}
func (e *ImmutableExpr) String() string {
return "immutable(" + e.Expr.String() + ")"
}

29
compiler/ast/import_expr.go
View file

@ -1,29 +0,0 @@
package ast
import (
"github.com/d5/tengo/compiler/source"
"github.com/d5/tengo/compiler/token"
)
// ImportExpr represents an import expression
type ImportExpr struct {
ModuleName string
Token token.Token
TokenPos source.Pos
}
func (e *ImportExpr) exprNode() {}
// Pos returns the position of first character belonging to the node.
func (e *ImportExpr) Pos() source.Pos {
return e.TokenPos
}
// End returns the position of first character immediately after the node.
func (e *ImportExpr) End() source.Pos {
return source.Pos(int(e.TokenPos) + 10 + len(e.ModuleName)) // import("moduleName")
}
func (e *ImportExpr) String() string {
return `import("` + e.ModuleName + `")"`
}

29
compiler/ast/inc_dec_stmt.go
View file

@ -1,29 +0,0 @@
package ast
import (
"github.com/d5/tengo/compiler/source"
"github.com/d5/tengo/compiler/token"
)
// IncDecStmt represents increment or decrement statement.
type IncDecStmt struct {
Expr Expr
Token token.Token
TokenPos source.Pos
}
func (s *IncDecStmt) stmtNode() {}
// Pos returns the position of first character belonging to the node.
func (s *IncDecStmt) Pos() source.Pos {
return s.Expr.Pos()
}
// End returns the position of first character immediately after the node.
func (s *IncDecStmt) End() source.Pos {
return source.Pos(int(s.TokenPos) + 2)
}
func (s *IncDecStmt) String() string {
return s.Expr.String() + s.Token.String()
}

32
compiler/ast/index_expr.go
View file

@ -1,32 +0,0 @@
package ast
import "github.com/d5/tengo/compiler/source"
// IndexExpr represents an index expression.
type IndexExpr struct {
Expr Expr
LBrack source.Pos
Index Expr
RBrack source.Pos
}
func (e *IndexExpr) exprNode() {}
// Pos returns the position of first character belonging to the node.
func (e *IndexExpr) Pos() source.Pos {
return e.Expr.Pos()
}
// End returns the position of first character immediately after the node.
func (e *IndexExpr) End() source.Pos {
return e.RBrack + 1
}
func (e *IndexExpr) String() string {
var index string
if e.Index != nil {
index = e.Index.String()
}
return e.Expr.String() + "[" + index + "]"
}

26
compiler/ast/int_lit.go
View file

@ -1,26 +0,0 @@
package ast
import "github.com/d5/tengo/compiler/source"
// IntLit represents an integer literal.
type IntLit struct {
Value int64
ValuePos source.Pos
Literal string
}
func (e *IntLit) exprNode() {}
// Pos returns the position of first character belonging to the node.
func (e *IntLit) Pos() source.Pos {
return e.ValuePos
}
// End returns the position of first character immediately after the node.
func (e *IntLit) End() source.Pos {
return source.Pos(int(e.ValuePos) + len(e.Literal))
}
func (e *IntLit) String() string {
return e.Literal
}

27
compiler/ast/map_element_lit.go
View file

@ -1,27 +0,0 @@
package ast
import "github.com/d5/tengo/compiler/source"
// MapElementLit represents a map element.
type MapElementLit struct {
Key string
KeyPos source.Pos
ColonPos source.Pos
Value Expr
}
func (e *MapElementLit) exprNode() {}
// Pos returns the position of first character belonging to the node.
func (e *MapElementLit) Pos() source.Pos {
return e.KeyPos
}
// End returns the position of first character immediately after the node.
func (e *MapElementLit) End() source.Pos {
return e.Value.End()
}
func (e *MapElementLit) String() string {
return e.Key + ": " + e.Value.String()
}

35
compiler/ast/map_lit.go
View file

@ -1,35 +0,0 @@
package ast
import (
"strings"
"github.com/d5/tengo/compiler/source"
)
// MapLit represents a map literal.
type MapLit struct {
LBrace source.Pos
Elements []*MapElementLit
RBrace source.Pos
}
func (e *MapLit) exprNode() {}
// Pos returns the position of first character belonging to the node.
func (e *MapLit) Pos() source.Pos {
return e.LBrace
}
// End returns the position of first character immediately after the node.
func (e *MapLit) End() source.Pos {
return e.RBrace + 1
}
func (e *MapLit) String() string {
var elements []string
for _, m := range e.Elements {
elements = append(elements, m.String())
}
return "{" + strings.Join(elements, ", ") + "}"
}

13
compiler/ast/node.go
View file

@ -1,13 +0,0 @@
package ast
import "github.com/d5/tengo/compiler/source"
// Node represents a node in the AST.
type Node interface {
// Pos returns the position of first character belonging to the node.
Pos() source.Pos
// End returns the position of first character immediately after the node.
End() source.Pos
// String returns a string representation of the node.
String() string
}

26
compiler/ast/paren_expr.go
View file

@ -1,26 +0,0 @@
package ast
import "github.com/d5/tengo/compiler/source"
// ParenExpr represents a parenthesis wrapped expression.
type ParenExpr struct {
Expr Expr
LParen source.Pos
RParen source.Pos
}
func (e *ParenExpr) exprNode() {}
// Pos returns the position of first character belonging to the node.
func (e *ParenExpr) Pos() source.Pos {
return e.LParen
}
// End returns the position of first character immediately after the node.
func (e *ParenExpr) End() source.Pos {
return e.RParen + 1
}
func (e *ParenExpr) String() string {
return "(" + e.Expr.String() + ")"
}

35
compiler/ast/return_stmt.go
View file

@ -1,35 +0,0 @@
package ast
import (
"github.com/d5/tengo/compiler/source"
)
// ReturnStmt represents a return statement.
type ReturnStmt struct {
ReturnPos source.Pos
Result Expr
}
func (s *ReturnStmt) stmtNode() {}
// Pos returns the position of first character belonging to the node.
func (s *ReturnStmt) Pos() source.Pos {
return s.ReturnPos
}
// End returns the position of first character immediately after the node.
func (s *ReturnStmt) End() source.Pos {
if s.Result != nil {
return s.Result.End()
}
return s.ReturnPos + 6
}
func (s *ReturnStmt) String() string {
if s.Result != nil {
return "return " + s.Result.String()
}
return "return"
}

25
compiler/ast/selector_expr.go
View file

@ -1,25 +0,0 @@
package ast
import "github.com/d5/tengo/compiler/source"
// SelectorExpr represents a selector expression.
type SelectorExpr struct {
Expr Expr
Sel Expr
}
func (e *SelectorExpr) exprNode() {}
// Pos returns the position of first character belonging to the node.
func (e *SelectorExpr) Pos() source.Pos {
return e.Expr.Pos()
}
// End returns the position of first character immediately after the node.
func (e *SelectorExpr) End() source.Pos {
return e.Sel.End()
}
func (e *SelectorExpr) String() string {
return e.Expr.String() + "." + e.Sel.String()
}

36
compiler/ast/slice_expr.go
View file

@ -1,36 +0,0 @@
package ast
import "github.com/d5/tengo/compiler/source"
// SliceExpr represents a slice expression.
type SliceExpr struct {
Expr Expr
LBrack source.Pos
Low Expr
High Expr
RBrack source.Pos
}
func (e *SliceExpr) exprNode() {}
// Pos returns the position of first character belonging to the node.
func (e *SliceExpr) Pos() source.Pos {
return e.Expr.Pos()
}
// End returns the position of first character immediately after the node.
func (e *SliceExpr) End() source.Pos {
return e.RBrack + 1
}
func (e *SliceExpr) String() string {
var low, high string
if e.Low != nil {
low = e.Low.String()
}
if e.High != nil {
high = e.High.String()
}
return e.Expr.String() + "[" + low + ":" + high + "]"
}

7
compiler/ast/stmt.go
View file

@ -1,7 +0,0 @@
package ast
// Stmt represents a statement in the AST.
type Stmt interface {
Node
stmtNode()
}

26
compiler/ast/string_lit.go
View file

@ -1,26 +0,0 @@
package ast
import "github.com/d5/tengo/compiler/source"
// StringLit represents a string literal.
type StringLit struct {
Value string
ValuePos source.Pos
Literal string
}
func (e *StringLit) exprNode() {}
// Pos returns the position of first character belonging to the node.
func (e *StringLit) Pos() source.Pos {
return e.ValuePos
}
// End returns the position of first character immediately after the node.
func (e *StringLit) End() source.Pos {
return source.Pos(int(e.ValuePos) + len(e.Literal))
}
func (e *StringLit) String() string {
return e.Literal
}

29
compiler/ast/unary_expr.go
View file

@ -1,29 +0,0 @@
package ast
import (
"github.com/d5/tengo/compiler/source"
"github.com/d5/tengo/compiler/token"
)
// UnaryExpr represents an unary operator expression.
type UnaryExpr struct {
Expr Expr
Token token.Token
TokenPos source.Pos
}
func (e *UnaryExpr) exprNode() {}
// Pos returns the position of first character belonging to the node.
func (e *UnaryExpr) Pos() source.Pos {
return e.Expr.Pos()
}
// End returns the position of first character immediately after the node.
func (e *UnaryExpr) End() source.Pos {
return e.Expr.End()
}
func (e *UnaryExpr) String() string {
return "(" + e.Token.String() + e.Expr.String() + ")"
}

24
compiler/ast/undefined_lit.go
View file

@ -1,24 +0,0 @@
package ast
import "github.com/d5/tengo/compiler/source"
// UndefinedLit represents an undefined literal.
type UndefinedLit struct {
TokenPos source.Pos
}
func (e *UndefinedLit) exprNode() {}
// Pos returns the position of first character belonging to the node.
func (e *UndefinedLit) Pos() source.Pos {
return e.TokenPos
}
// End returns the position of first character immediately after the node.
func (e *UndefinedLit) End() source.Pos {
return e.TokenPos + 9 // len(undefined) == 9
}
func (e *UndefinedLit) String() string {
return "undefined"
}

90
compiler/bytecode.go
View file

@ -1,90 +0,0 @@
package compiler
import (
"encoding/gob"
"fmt"
"io"
"reflect"
"github.com/d5/tengo/compiler/source"
"github.com/d5/tengo/objects"
)
// Bytecode is a compiled instructions and constants.
type Bytecode struct {
FileSet *source.FileSet
MainFunction *objects.CompiledFunction
Constants []objects.Object
}
// Encode writes Bytecode data to the writer.
func (b *Bytecode) Encode(w io.Writer) error {
enc := gob.NewEncoder(w)
if err := enc.Encode(b.FileSet); err != nil {
return err
}
if err := enc.Encode(b.MainFunction); err != nil {
return err
}
// constants
return enc.Encode(b.Constants)
}
// CountObjects returns the number of objects found in Constants.
func (b *Bytecode) CountObjects() int {
n := 0
for _, c := range b.Constants {
n += objects.CountObjects(c)
}
return n
}
// FormatInstructions returns human readable string representations of
// compiled instructions.
func (b *Bytecode) FormatInstructions() []string {
return FormatInstructions(b.MainFunction.Instructions, 0)
}
// FormatConstants returns human readable string representations of
// compiled constants.
func (b *Bytecode) FormatConstants() (output []string) {
for cidx, cn := range b.Constants {
switch cn := cn.(type) {
case *objects.CompiledFunction:
output = append(output, fmt.Sprintf("[% 3d] (Compiled Function|%p)", cidx, &cn))
for _, l := range FormatInstructions(cn.Instructions, 0) {
output = append(output, fmt.Sprintf(" %s", l))
}
default:
output = append(output, fmt.Sprintf("[% 3d] %s (%s|%p)", cidx, cn, reflect.TypeOf(cn).Elem().Name(), &cn))
}
}
return
}
func init() {
gob.Register(&source.FileSet{})
gob.Register(&source.File{})
gob.Register(&objects.Array{})
gob.Register(&objects.Bool{})
gob.Register(&objects.Bytes{})
gob.Register(&objects.Char{})
gob.Register(&objects.Closure{})
gob.Register(&objects.CompiledFunction{})
gob.Register(&objects.Error{})
gob.Register(&objects.Float{})
gob.Register(&objects.ImmutableArray{})
gob.Register(&objects.ImmutableMap{})
gob.Register(&objects.Int{})
gob.Register(&objects.Map{})
gob.Register(&objects.String{})
gob.Register(&objects.Time{})
gob.Register(&objects.Undefined{})
gob.Register(&objects.UserFunction{})
}

97
compiler/bytecode_decode.go
View file

@ -1,97 +0,0 @@
package compiler
import (
"encoding/gob"
"fmt"
"io"
"github.com/d5/tengo/objects"
)
// Decode reads Bytecode data from the reader.
func (b *Bytecode) Decode(r io.Reader, modules *objects.ModuleMap) error {
if modules == nil {
modules = objects.NewModuleMap()
}
dec := gob.NewDecoder(r)
if err := dec.Decode(&b.FileSet); err != nil {
return err
}
// TODO: files in b.FileSet.File does not have their 'set' field properly set to b.FileSet
// as it's private field and not serialized by gob encoder/decoder.
if err := dec.Decode(&b.MainFunction); err != nil {
return err
}
if err := dec.Decode(&b.Constants); err != nil {
return err
}
for i, v := range b.Constants {
fv, err := fixDecoded(v, modules)
if err != nil {
return err
}
b.Constants[i] = fv
}
return nil
}
func fixDecoded(o objects.Object, modules *objects.ModuleMap) (objects.Object, error) {
switch o := o.(type) {
case *objects.Bool:
if o.IsFalsy() {
return objects.FalseValue, nil
}
return objects.TrueValue, nil
case *objects.Undefined:
return objects.UndefinedValue, nil
case *objects.Array:
for i, v := range o.Value {
fv, err := fixDecoded(v, modules)
if err != nil {
return nil, err
}
o.Value[i] = fv
}
case *objects.ImmutableArray:
for i, v := range o.Value {
fv, err := fixDecoded(v, modules)
if err != nil {
return nil, err
}
o.Value[i] = fv
}
case *objects.Map:
for k, v := range o.Value {
fv, err := fixDecoded(v, modules)
if err != nil {
return nil, err
}
o.Value[k] = fv
}
case *objects.ImmutableMap:
modName := moduleName(o)
if mod := modules.GetBuiltinModule(modName); mod != nil {
return mod.AsImmutableMap(modName), nil
}
for k, v := range o.Value {
// encoding of user function not supported
if _, isUserFunction := v.(*objects.UserFunction); isUserFunction {
return nil, fmt.Errorf("user function not decodable")
}
fv, err := fixDecoded(v, modules)
if err != nil {
return nil, err
}
o.Value[k] = fv
}
}
return o, nil
}

129
compiler/bytecode_optimize.go
View file

@ -1,129 +0,0 @@
package compiler
import (
"fmt"
"github.com/d5/tengo/objects"
)
// RemoveDuplicates finds and remove the duplicate values in Constants.
// Note this function mutates Bytecode.
func (b *Bytecode) RemoveDuplicates() {
var deduped []objects.Object
indexMap := make(map[int]int) // mapping from old constant index to new index
ints := make(map[int64]int)
strings := make(map[string]int)
floats := make(map[float64]int)
chars := make(map[rune]int)
immutableMaps := make(map[string]int) // for modules
for curIdx, c := range b.Constants {
switch c := c.(type) {
case *objects.CompiledFunction:
// add to deduped list
indexMap[curIdx] = len(deduped)
deduped = append(deduped, c)
case *objects.ImmutableMap:
modName := moduleName(c)
newIdx, ok := immutableMaps[modName]
if modName != "" && ok {
indexMap[curIdx] = newIdx
} else {
newIdx = len(deduped)
immutableMaps[modName] = newIdx
indexMap[curIdx] = newIdx
deduped = append(deduped, c)
}
case *objects.Int:
if newIdx, ok := ints[c.Value]; ok {
indexMap[curIdx] = newIdx
} else {
newIdx = len(deduped)
ints[c.Value] = newIdx
indexMap[curIdx] = newIdx
deduped = append(deduped, c)
}
case *objects.String:
if newIdx, ok := strings[c.Value]; ok {
indexMap[curIdx] = newIdx
} else {
newIdx = len(deduped)
strings[c.Value] = newIdx
indexMap[curIdx] = newIdx
deduped = append(deduped, c)
}
case *objects.Float:
if newIdx, ok := floats[c.Value]; ok {
indexMap[curIdx] = newIdx
} else {
newIdx = len(deduped)
floats[c.Value] = newIdx
indexMap[curIdx] = newIdx
deduped = append(deduped, c)
}
case *objects.Char:
if newIdx, ok := chars[c.Value]; ok {
indexMap[curIdx] = newIdx
} else {
newIdx = len(deduped)
chars[c.Value] = newIdx
indexMap[curIdx] = newIdx
deduped = append(deduped, c)
}
default:
panic(fmt.Errorf("unsupported top-level constant type: %s", c.TypeName()))
}
}
// replace with de-duplicated constants
b.Constants = deduped
// update CONST instructions with new indexes
// main function
updateConstIndexes(b.MainFunction.Instructions, indexMap)
// other compiled functions in constants
for _, c := range b.Constants {
switch c := c.(type) {
case *objects.CompiledFunction:
updateConstIndexes(c.Instructions, indexMap)
}
}
}
func updateConstIndexes(insts []byte, indexMap map[int]int) {
i := 0
for i < len(insts) {
op := insts[i]
numOperands := OpcodeOperands[op]
_, read := ReadOperands(numOperands, insts[i+1:])
switch op {
case OpConstant:
curIdx := int(insts[i+2]) | int(insts[i+1])<<8
newIdx, ok := indexMap[curIdx]
if !ok {
panic(fmt.Errorf("constant index not found: %d", curIdx))
}
copy(insts[i:], MakeInstruction(op, newIdx))
case OpClosure:
curIdx := int(insts[i+2]) | int(insts[i+1])<<8
numFree := int(insts[i+3])
newIdx, ok := indexMap[curIdx]
if !ok {
panic(fmt.Errorf("constant index not found: %d", curIdx))
}
copy(insts[i:], MakeInstruction(op, newIdx, numFree))
}
i += 1 + read
}
}
func moduleName(mod *objects.ImmutableMap) string {
if modName, ok := mod.Value["__module_name__"].(*objects.String); ok {
return modName.Value
}
return ""
}

289
compiler/bytecode_test.go
View file

@ -1,289 +0,0 @@
package compiler_test
import (
"bytes"
"testing"
"time"
"github.com/d5/tengo/assert"
"github.com/d5/tengo/compiler"
"github.com/d5/tengo/compiler/source"
"github.com/d5/tengo/objects"
)
type srcfile struct {
name string
size int
}
func TestBytecode(t *testing.T) {
testBytecodeSerialization(t, bytecode(concat(), objectsArray()))
testBytecodeSerialization(t, bytecode(
concat(), objectsArray(
&objects.Char{Value: 'y'},
&objects.Float{Value: 93.11},
compiledFunction(1, 0,
compiler.MakeInstruction(compiler.OpConstant, 3),
compiler.MakeInstruction(compiler.OpSetLocal, 0),
compiler.MakeInstruction(compiler.OpGetGlobal, 0),
compiler.MakeInstruction(compiler.OpGetFree, 0)),
&objects.Float{Value: 39.2},
&objects.Int{Value: 192},
&objects.String{Value: "bar"})))
testBytecodeSerialization(t, bytecodeFileSet(
concat(
compiler.MakeInstruction(compiler.OpConstant, 0),
compiler.MakeInstruction(compiler.OpSetGlobal, 0),
compiler.MakeInstruction(compiler.OpConstant, 6),
compiler.MakeInstruction(compiler.OpPop)),
objectsArray(
&objects.Int{Value: 55},
&objects.Int{Value: 66},
&objects.Int{Value: 77},
&objects.Int{Value: 88},
&objects.ImmutableMap{
Value: map[string]objects.Object{
"array": &objects.ImmutableArray{
Value: []objects.Object{
&objects.Int{Value: 1},
&objects.Int{Value: 2},
&objects.Int{Value: 3},
objects.TrueValue,
objects.FalseValue,
objects.UndefinedValue,
},
},
"true": objects.TrueValue,
"false": objects.FalseValue,
"bytes": &objects.Bytes{Value: make([]byte, 16)},
"char": &objects.Char{Value: 'Y'},
"error": &objects.Error{Value: &objects.String{Value: "some error"}},
"float": &objects.Float{Value: -19.84},
"immutable_array": &objects.ImmutableArray{
Value: []objects.Object{
&objects.Int{Value: 1},
&objects.Int{Value: 2},
&objects.Int{Value: 3},
objects.TrueValue,
objects.FalseValue,
objects.UndefinedValue,
},
},
"immutable_map": &objects.ImmutableMap{
Value: map[string]objects.Object{
"a": &objects.Int{Value: 1},
"b": &objects.Int{Value: 2},
"c": &objects.Int{Value: 3},
"d": objects.TrueValue,
"e": objects.FalseValue,
"f": objects.UndefinedValue,
},
},
"int": &objects.Int{Value: 91},
"map": &objects.Map{
Value: map[string]objects.Object{
"a": &objects.Int{Value: 1},
"b": &objects.Int{Value: 2},
"c": &objects.Int{Value: 3},
"d": objects.TrueValue,
"e": objects.FalseValue,
"f": objects.UndefinedValue,
},
},
"string": &objects.String{Value: "foo bar"},
"time": &objects.Time{Value: time.Now()},
"undefined": objects.UndefinedValue,
},
},
compiledFunction(1, 0,
compiler.MakeInstruction(compiler.OpConstant, 3),
compiler.MakeInstruction(compiler.OpSetLocal, 0),
compiler.MakeInstruction(compiler.OpGetGlobal, 0),
compiler.MakeInstruction(compiler.OpGetFree, 0),
compiler.MakeInstruction(compiler.OpBinaryOp, 11),
compiler.MakeInstruction(compiler.OpGetFree, 1),
compiler.MakeInstruction(compiler.OpBinaryOp, 11),
compiler.MakeInstruction(compiler.OpGetLocal, 0),
compiler.MakeInstruction(compiler.OpBinaryOp, 11),
compiler.MakeInstruction(compiler.OpReturn, 1)),
compiledFunction(1, 0,
compiler.MakeInstruction(compiler.OpConstant, 2),
compiler.MakeInstruction(compiler.OpSetLocal, 0),
compiler.MakeInstruction(compiler.OpGetFree, 0),
compiler.MakeInstruction(compiler.OpGetLocal, 0),
compiler.MakeInstruction(compiler.OpClosure, 4, 2),
compiler.MakeInstruction(compiler.OpReturn, 1)),
compiledFunction(1, 0,
compiler.MakeInstruction(compiler.OpConstant, 1),
compiler.MakeInstruction(compiler.OpSetLocal, 0),
compiler.MakeInstruction(compiler.OpGetLocal, 0),
compiler.MakeInstruction(compiler.OpClosure, 5, 1),
compiler.MakeInstruction(compiler.OpReturn, 1))),
fileSet(srcfile{name: "file1", size: 100}, srcfile{name: "file2", size: 200})))
}
func TestBytecode_RemoveDuplicates(t *testing.T) {
testBytecodeRemoveDuplicates(t,
bytecode(
concat(), objectsArray(
&objects.Char{Value: 'y'},
&objects.Float{Value: 93.11},
compiledFunction(1, 0,
compiler.MakeInstruction(compiler.OpConstant, 3),
compiler.MakeInstruction(compiler.OpSetLocal, 0),
compiler.MakeInstruction(compiler.OpGetGlobal, 0),
compiler.MakeInstruction(compiler.OpGetFree, 0)),
&objects.Float{Value: 39.2},
&objects.Int{Value: 192},
&objects.String{Value: "bar"})),
bytecode(
concat(), objectsArray(
&objects.Char{Value: 'y'},
&objects.Float{Value: 93.11},
compiledFunction(1, 0,
compiler.MakeInstruction(compiler.OpConstant, 3),
compiler.MakeInstruction(compiler.OpSetLocal, 0),
compiler.MakeInstruction(compiler.OpGetGlobal, 0),
compiler.MakeInstruction(compiler.OpGetFree, 0)),
&objects.Float{Value: 39.2},
&objects.Int{Value: 192},
&objects.String{Value: "bar"})))
testBytecodeRemoveDuplicates(t,
bytecode(
concat(
compiler.MakeInstruction(compiler.OpConstant, 0),
compiler.MakeInstruction(compiler.OpConstant, 1),
compiler.MakeInstruction(compiler.OpConstant, 2),
compiler.MakeInstruction(compiler.OpConstant, 3),
compiler.MakeInstruction(compiler.OpConstant, 4),
compiler.MakeInstruction(compiler.OpConstant, 5),
compiler.MakeInstruction(compiler.OpConstant, 6),
compiler.MakeInstruction(compiler.OpConstant, 7),
compiler.MakeInstruction(compiler.OpConstant, 8),
compiler.MakeInstruction(compiler.OpClosure, 4, 1)),
objectsArray(
&objects.Int{Value: 1},
&objects.Float{Value: 2.0},
&objects.Char{Value: '3'},
&objects.String{Value: "four"},
compiledFunction(1, 0,
compiler.MakeInstruction(compiler.OpConstant, 3),
compiler.MakeInstruction(compiler.OpConstant, 7),
compiler.MakeInstruction(compiler.OpSetLocal, 0),
compiler.MakeInstruction(compiler.OpGetGlobal, 0),
compiler.MakeInstruction(compiler.OpGetFree, 0)),
&objects.Int{Value: 1},
&objects.Float{Value: 2.0},
&objects.Char{Value: '3'},
&objects.String{Value: "four"})),
bytecode(
concat(
compiler.MakeInstruction(compiler.OpConstant, 0),
compiler.MakeInstruction(compiler.OpConstant, 1),
compiler.MakeInstruction(compiler.OpConstant, 2),
compiler.MakeInstruction(compiler.OpConstant, 3),
compiler.MakeInstruction(compiler.OpConstant, 4),
compiler.MakeInstruction(compiler.OpConstant, 0),
compiler.MakeInstruction(compiler.OpConstant, 1),
compiler.MakeInstruction(compiler.OpConstant, 2),
compiler.MakeInstruction(compiler.OpConstant, 3),
compiler.MakeInstruction(compiler.OpClosure, 4, 1)),
objectsArray(
&objects.Int{Value: 1},
&objects.Float{Value: 2.0},
&objects.Char{Value: '3'},
&objects.String{Value: "four"},
compiledFunction(1, 0,
compiler.MakeInstruction(compiler.OpConstant, 3),
compiler.MakeInstruction(compiler.OpConstant, 2),
compiler.MakeInstruction(compiler.OpSetLocal, 0),
compiler.MakeInstruction(compiler.OpGetGlobal, 0),
compiler.MakeInstruction(compiler.OpGetFree, 0)))))
testBytecodeRemoveDuplicates(t,
bytecode(
concat(
compiler.MakeInstruction(compiler.OpConstant, 0),
compiler.MakeInstruction(compiler.OpConstant, 1),
compiler.MakeInstruction(compiler.OpConstant, 2),
compiler.MakeInstruction(compiler.OpConstant, 3),
compiler.MakeInstruction(compiler.OpConstant, 4)),
objectsArray(
&objects.Int{Value: 1},
&objects.Int{Value: 2},
&objects.Int{Value: 3},
&objects.Int{Value: 1},
&objects.Int{Value: 3})),
bytecode(
concat(
compiler.MakeInstruction(compiler.OpConstant, 0),
compiler.MakeInstruction(compiler.OpConstant, 1),
compiler.MakeInstruction(compiler.OpConstant, 2),
compiler.MakeInstruction(compiler.OpConstant, 0),
compiler.MakeInstruction(compiler.OpConstant, 2)),
objectsArray(
&objects.Int{Value: 1},
&objects.Int{Value: 2},
&objects.Int{Value: 3})))
}
func TestBytecode_CountObjects(t *testing.T) {
b := bytecode(
concat(),
objectsArray(
&objects.Int{Value: 55},
&objects.Int{Value: 66},
&objects.Int{Value: 77},
&objects.Int{Value: 88},
compiledFunction(1, 0,
compiler.MakeInstruction(compiler.OpConstant, 3),
compiler.MakeInstruction(compiler.OpReturn, 1)),
compiledFunction(1, 0,
compiler.MakeInstruction(compiler.OpConstant, 2),
compiler.MakeInstruction(compiler.OpReturn, 1)),
compiledFunction(1, 0,
compiler.MakeInstruction(compiler.OpConstant, 1),
compiler.MakeInstruction(compiler.OpReturn, 1))))
assert.Equal(t, 7, b.CountObjects())
}
func fileSet(files ...srcfile) *source.FileSet {
fileSet := source.NewFileSet()
for _, f := range files {
fileSet.AddFile(f.name, -1, f.size)
}
return fileSet
}
func bytecodeFileSet(instructions []byte, constants []objects.Object, fileSet *source.FileSet) *compiler.Bytecode {
return &compiler.Bytecode{
FileSet: fileSet,
MainFunction: &objects.CompiledFunction{Instructions: instructions},
Constants: constants,
}
}
func testBytecodeRemoveDuplicates(t *testing.T, input, expected *compiler.Bytecode) {
input.RemoveDuplicates()
assert.Equal(t, expected.FileSet, input.FileSet)
assert.Equal(t, expected.MainFunction, input.MainFunction)
assert.Equal(t, expected.Constants, input.Constants)
}
func testBytecodeSerialization(t *testing.T, b *compiler.Bytecode) {
var buf bytes.Buffer
err := b.Encode(&buf)
assert.NoError(t, err)
r := &compiler.Bytecode{}
err = r.Decode(bytes.NewReader(buf.Bytes()), nil)
assert.NoError(t, err)
assert.Equal(t, b.FileSet, r.FileSet)
assert.Equal(t, b.MainFunction, r.MainFunction)
assert.Equal(t, b.Constants, r.Constants)
}

11
compiler/compilation_scope.go
View file

@ -1,11 +0,0 @@
package compiler
import "github.com/d5/tengo/compiler/source"
// CompilationScope represents a compiled instructions
// and the last two instructions that were emitted.
type CompilationScope struct {
instructions []byte
symbolInit map[string]bool
sourceMap map[int]source.Pos
}

846
compiler/compiler.go
View file

@ -1,846 +0,0 @@
package compiler
import (
"fmt"
"io"
"io/ioutil"
"path/filepath"
"reflect"
"strings"
"github.com/d5/tengo"
"github.com/d5/tengo/compiler/ast"
"github.com/d5/tengo/compiler/source"
"github.com/d5/tengo/compiler/token"
"github.com/d5/tengo/objects"
)
// Compiler compiles the AST into a bytecode.
type Compiler struct {
file *source.File
parent *Compiler
modulePath string
constants []objects.Object
symbolTable *SymbolTable
scopes []CompilationScope
scopeIndex int
modules *objects.ModuleMap
compiledModules map[string]*objects.CompiledFunction
allowFileImport bool
loops []*Loop
loopIndex int
trace io.Writer
indent int
}
// NewCompiler creates a Compiler.
func NewCompiler(file *source.File, symbolTable *SymbolTable, constants []objects.Object, modules *objects.ModuleMap, trace io.Writer) *Compiler {
mainScope := CompilationScope{
symbolInit: make(map[string]bool),
sourceMap: make(map[int]source.Pos),
}
// symbol table
if symbolTable == nil {
symbolTable = NewSymbolTable()
}
// add builtin functions to the symbol table
for idx, fn := range objects.Builtins {
symbolTable.DefineBuiltin(idx, fn.Name)
}
// builtin modules
if modules == nil {
modules = objects.NewModuleMap()
}
return &Compiler{
file: file,
symbolTable: symbolTable,
constants: constants,
scopes: []CompilationScope{mainScope},
scopeIndex: 0,
loopIndex: -1,
trace: trace,
modules: modules,
compiledModules: make(map[string]*objects.CompiledFunction),
}
}
// Compile compiles the AST node.
func (c *Compiler) Compile(node ast.Node) error {
if c.trace != nil {
if node != nil {
defer un(trace(c, fmt.Sprintf("%s (%s)", node.String(), reflect.TypeOf(node).Elem().Name())))
} else {
defer un(trace(c, "<nil>"))
}
}
switch node := node.(type) {
case *ast.File:
for _, stmt := range node.Stmts {
if err := c.Compile(stmt); err != nil {
return err
}
}
case *ast.ExprStmt:
if err := c.Compile(node.Expr); err != nil {
return err
}
c.emit(node, OpPop)
case *ast.IncDecStmt:
op := token.AddAssign
if node.Token == token.Dec {
op = token.SubAssign
}
return c.compileAssign(node, []ast.Expr{node.Expr}, []ast.Expr{&ast.IntLit{Value: 1}}, op)
case *ast.ParenExpr:
if err := c.Compile(node.Expr); err != nil {
return err
}
case *ast.BinaryExpr:
if node.Token == token.LAnd || node.Token == token.LOr {
return c.compileLogical(node)
}
if node.Token == token.Less {
if err := c.Compile(node.RHS); err != nil {
return err
}
if err := c.Compile(node.LHS); err != nil {
return err
}
c.emit(node, OpBinaryOp, int(token.Greater))
return nil
} else if node.Token == token.LessEq {
if err := c.Compile(node.RHS); err != nil {
return err
}
if err := c.Compile(node.LHS); err != nil {
return err
}
c.emit(node, OpBinaryOp, int(token.GreaterEq))
return nil
}
if err := c.Compile(node.LHS); err != nil {
return err
}
if err := c.Compile(node.RHS); err != nil {
return err
}
switch node.Token {
case token.Add:
c.emit(node, OpBinaryOp, int(token.Add))
case token.Sub:
c.emit(node, OpBinaryOp, int(token.Sub))
case token.Mul:
c.emit(node, OpBinaryOp, int(token.Mul))
case token.Quo:
c.emit(node, OpBinaryOp, int(token.Quo))
case token.Rem:
c.emit(node, OpBinaryOp, int(token.Rem))
case token.Greater:
c.emit(node, OpBinaryOp, int(token.Greater))
case token.GreaterEq:
c.emit(node, OpBinaryOp, int(token.GreaterEq))
case token.Equal:
c.emit(node, OpEqual)
case token.NotEqual:
c.emit(node, OpNotEqual)
case token.And:
c.emit(node, OpBinaryOp, int(token.And))
case token.Or:
c.emit(node, OpBinaryOp, int(token.Or))
case token.Xor:
c.emit(node, OpBinaryOp, int(token.Xor))
case token.AndNot:
c.emit(node, OpBinaryOp, int(token.AndNot))
case token.Shl:
c.emit(node, OpBinaryOp, int(token.Shl))
case token.Shr:
c.emit(node, OpBinaryOp, int(token.Shr))
default:
return c.errorf(node, "invalid binary operator: %s", node.Token.String())
}
case *ast.IntLit:
c.emit(node, OpConstant, c.addConstant(&objects.Int{Value: node.Value}))
case *ast.FloatLit:
c.emit(node, OpConstant, c.addConstant(&objects.Float{Value: node.Value}))
case *ast.BoolLit:
if node.Value {
c.emit(node, OpTrue)
} else {
c.emit(node, OpFalse)
}
case *ast.StringLit:
if len(node.Value) > tengo.MaxStringLen {
return c.error(node, objects.ErrStringLimit)
}
c.emit(node, OpConstant, c.addConstant(&objects.String{Value: node.Value}))
case *ast.CharLit:
c.emit(node, OpConstant, c.addConstant(&objects.Char{Value: node.Value}))
case *ast.UndefinedLit:
c.emit(node, OpNull)
case *ast.UnaryExpr:
if err := c.Compile(node.Expr); err != nil {
return err
}
switch node.Token {
case token.Not:
c.emit(node, OpLNot)
case token.Sub:
c.emit(node, OpMinus)
case token.Xor:
c.emit(node, OpBComplement)
case token.Add:
// do nothing?
default:
return c.errorf(node, "invalid unary operator: %s", node.Token.String())
}
case *ast.IfStmt:
// open new symbol table for the statement
c.symbolTable = c.symbolTable.Fork(true)
defer func() {
c.symbolTable = c.symbolTable.Parent(false)
}()
if node.Init != nil {
if err := c.Compile(node.Init); err != nil {
return err
}
}
if err := c.Compile(node.Cond); err != nil {
return err
}
// first jump placeholder
jumpPos1 := c.emit(node, OpJumpFalsy, 0)
if err := c.Compile(node.Body); err != nil {
return err
}
if node.Else != nil {
// second jump placeholder
jumpPos2 := c.emit(node, OpJump, 0)
// update first jump offset
curPos := len(c.currentInstructions())
c.changeOperand(jumpPos1, curPos)
if err := c.Compile(node.Else); err != nil {
return err
}
// update second jump offset
curPos = len(c.currentInstructions())
c.changeOperand(jumpPos2, curPos)
} else {
// update first jump offset
curPos := len(c.currentInstructions())
c.changeOperand(jumpPos1, curPos)
}
case *ast.ForStmt:
return c.compileForStmt(node)
case *ast.ForInStmt:
return c.compileForInStmt(node)
case *ast.BranchStmt:
if node.Token == token.Break {
curLoop := c.currentLoop()
if curLoop == nil {
return c.errorf(node, "break not allowed outside loop")
}
pos := c.emit(node, OpJump, 0)
curLoop.Breaks = append(curLoop.Breaks, pos)
} else if node.Token == token.Continue {
curLoop := c.currentLoop()
if curLoop == nil {
return c.errorf(node, "continue not allowed outside loop")
}
pos := c.emit(node, OpJump, 0)
curLoop.Continues = append(curLoop.Continues, pos)
} else {
panic(fmt.Errorf("invalid branch statement: %s", node.Token.String()))
}
case *ast.BlockStmt:
if len(node.Stmts) == 0 {
return nil
}
c.symbolTable = c.symbolTable.Fork(true)
defer func() {
c.symbolTable = c.symbolTable.Parent(false)
}()
for _, stmt := range node.Stmts {
if err := c.Compile(stmt); err != nil {
return err
}
}
case *ast.AssignStmt:
if err := c.compileAssign(node, node.LHS, node.RHS, node.Token); err != nil {
return err
}
case *ast.Ident:
symbol, _, ok := c.symbolTable.Resolve(node.Name)
if !ok {
return c.errorf(node, "unresolved reference '%s'", node.Name)
}
switch symbol.Scope {
case ScopeGlobal:
c.emit(node, OpGetGlobal, symbol.Index)
case ScopeLocal:
c.emit(node, OpGetLocal, symbol.Index)
case ScopeBuiltin:
c.emit(node, OpGetBuiltin, symbol.Index)
case ScopeFree:
c.emit(node, OpGetFree, symbol.Index)
}
case *ast.ArrayLit:
for _, elem := range node.Elements {
if err := c.Compile(elem); err != nil {
return err
}
}
c.emit(node, OpArray, len(node.Elements))
case *ast.MapLit:
for _, elt := range node.Elements {
// key
if len(elt.Key) > tengo.MaxStringLen {
return c.error(node, objects.ErrStringLimit)
}
c.emit(node, OpConstant, c.addConstant(&objects.String{Value: elt.Key}))
// value
if err := c.Compile(elt.Value); err != nil {
return err
}
}
c.emit(node, OpMap, len(node.Elements)*2)
case *ast.SelectorExpr: // selector on RHS side
if err := c.Compile(node.Expr); err != nil {
return err
}
if err := c.Compile(node.Sel); err != nil {
return err
}
c.emit(node, OpIndex)
case *ast.IndexExpr:
if err := c.Compile(node.Expr); err != nil {
return err
}
if err := c.Compile(node.Index); err != nil {
return err
}
c.emit(node, OpIndex)
case *ast.SliceExpr:
if err := c.Compile(node.Expr); err != nil {
return err
}
if node.Low != nil {
if err := c.Compile(node.Low); err != nil {
return err
}
} else {
c.emit(node, OpNull)
}
if node.High != nil {
if err := c.Compile(node.High); err != nil {
return err
}
} else {
c.emit(node, OpNull)
}
c.emit(node, OpSliceIndex)
case *ast.FuncLit:
c.enterScope()
for _, p := range node.Type.Params.List {
s := c.symbolTable.Define(p.Name)
// function arguments is not assigned directly.
s.LocalAssigned = true
}
if err := c.Compile(node.Body); err != nil {
return err
}
// code optimization
c.optimizeFunc(node)
freeSymbols := c.symbolTable.FreeSymbols()
numLocals := c.symbolTable.MaxSymbols()
instructions, sourceMap := c.leaveScope()
for _, s := range freeSymbols {
switch s.Scope {
case ScopeLocal:
if !s.LocalAssigned {
// Here, the closure is capturing a local variable that's not yet assigned its value.
// One example is a local recursive function:
//
// func() {
// foo := func(x) {
// // ..
// return foo(x-1)
// }
// }
//
// which translate into
//
// 0000 GETL 0
// 0002 CLOSURE ? 1
// 0006 DEFL 0
//
// . So the local variable (0) is being captured before it's assigned the value.
//
// Solution is to transform the code into something like this:
//
// func() {
// foo := undefined
// foo = func(x) {
// // ..
// return foo(x-1)
// }
// }
//
// that is equivalent to
//
// 0000 NULL
// 0001 DEFL 0
// 0003 GETL 0
// 0005 CLOSURE ? 1
// 0009 SETL 0
//
c.emit(node, OpNull)
c.emit(node, OpDefineLocal, s.Index)
s.LocalAssigned = true
}
c.emit(node, OpGetLocalPtr, s.Index)
case ScopeFree:
c.emit(node, OpGetFreePtr, s.Index)
}
}
compiledFunction := &objects.CompiledFunction{
Instructions: instructions,
NumLocals: numLocals,
NumParameters: len(node.Type.Params.List),
VarArgs: node.Type.Params.VarArgs,
SourceMap: sourceMap,
}
if len(freeSymbols) > 0 {
c.emit(node, OpClosure, c.addConstant(compiledFunction), len(freeSymbols))
} else {
c.emit(node, OpConstant, c.addConstant(compiledFunction))
}
case *ast.ReturnStmt:
if c.symbolTable.Parent(true) == nil {
// outside the function
return c.errorf(node, "return not allowed outside function")
}
if node.Result == nil {
c.emit(node, OpReturn, 0)
} else {
if err := c.Compile(node.Result); err != nil {
return err
}
c.emit(node, OpReturn, 1)
}
case *ast.CallExpr:
if err := c.Compile(node.Func); err != nil {
return err
}
for _, arg := range node.Args {
if err := c.Compile(arg); err != nil {
return err
}
}
c.emit(node, OpCall, len(node.Args))
case *ast.ImportExpr:
if node.ModuleName == "" {
return c.errorf(node, "empty module name")
}
if mod := c.modules.Get(node.ModuleName); mod != nil {
v, err := mod.Import(node.ModuleName)
if err != nil {
return err
}
switch v := v.(type) {
case []byte: // module written in Tengo
compiled, err := c.compileModule(node, node.ModuleName, node.ModuleName, v)
if err != nil {
return err
}
c.emit(node, OpConstant, c.addConstant(compiled))
c.emit(node, OpCall, 0)
case objects.Object: // builtin module
c.emit(node, OpConstant, c.addConstant(v))
default:
panic(fmt.Errorf("invalid import value type: %T", v))
}
} else if c.allowFileImport {
moduleName := node.ModuleName
if !strings.HasSuffix(moduleName, ".tengo") {
moduleName += ".tengo"
}
modulePath, err := filepath.Abs(moduleName)
if err != nil {
return c.errorf(node, "module file path error: %s", err.Error())
}
if err := c.checkCyclicImports(node, modulePath); err != nil {
return err
}
moduleSrc, err := ioutil.ReadFile(moduleName)
if err != nil {
return c.errorf(node, "module file read error: %s", err.Error())
}
compiled, err := c.compileModule(node, moduleName, modulePath, moduleSrc)
if err != nil {
return err
}
c.emit(node, OpConstant, c.addConstant(compiled))
c.emit(node, OpCall, 0)
} else {
return c.errorf(node, "module '%s' not found", node.ModuleName)
}
case *ast.ExportStmt:
// export statement must be in top-level scope
if c.scopeIndex != 0 {
return c.errorf(node, "export not allowed inside function")
}
// export statement is simply ignore when compiling non-module code
if c.parent == nil {
break
}
if err := c.Compile(node.Result); err != nil {
return err
}
c.emit(node, OpImmutable)
c.emit(node, OpReturn, 1)
case *ast.ErrorExpr:
if err := c.Compile(node.Expr); err != nil {
return err
}
c.emit(node, OpError)
case *ast.ImmutableExpr:
if err := c.Compile(node.Expr); err != nil {
return err
}
c.emit(node, OpImmutable)
case *ast.CondExpr:
if err := c.Compile(node.Cond); err != nil {
return err
}
// first jump placeholder
jumpPos1 := c.emit(node, OpJumpFalsy, 0)
if err := c.Compile(node.True); err != nil {
return err
}
// second jump placeholder
jumpPos2 := c.emit(node, OpJump, 0)
// update first jump offset
curPos := len(c.currentInstructions())
c.changeOperand(jumpPos1, curPos)
if err := c.Compile(node.False); err != nil {
return err
}
// update second jump offset
curPos = len(c.currentInstructions())
c.changeOperand(jumpPos2, curPos)
}
return nil
}
// Bytecode returns a compiled bytecode.
func (c *Compiler) Bytecode() *Bytecode {
return &Bytecode{
FileSet: c.file.Set(),
MainFunction: &objects.CompiledFunction{
Instructions: c.currentInstructions(),
SourceMap: c.currentSourceMap(),
},
Constants: c.constants,
}
}
// EnableFileImport enables or disables module loading from local files.
// Local file modules are disabled by default.
func (c *Compiler) EnableFileImport(enable bool) {
c.allowFileImport = enable
}
func (c *Compiler) fork(file *source.File, modulePath string, symbolTable *SymbolTable) *Compiler {
child := NewCompiler(file, symbolTable, nil, c.modules, c.trace)
child.modulePath = modulePath // module file path
child.parent = c // parent to set to current compiler
return child
}
func (c *Compiler) error(node ast.Node, err error) error {
return &Error{
fileSet: c.file.Set(),
node: node,
error: err,
}
}
func (c *Compiler) errorf(node ast.Node, format string, args ...interface{}) error {
return &Error{
fileSet: c.file.Set(),
node: node,
error: fmt.Errorf(format, args...),
}
}
func (c *Compiler) addConstant(o objects.Object) int {
if c.parent != nil {
// module compilers will use their parent's constants array
return c.parent.addConstant(o)
}
c.constants = append(c.constants, o)
if c.trace != nil {
c.printTrace(fmt.Sprintf("CONST %04d %s", len(c.constants)-1, o))
}
return len(c.constants) - 1
}
func (c *Compiler) addInstruction(b []byte) int {
posNewIns := len(c.currentInstructions())
c.scopes[c.scopeIndex].instructions = append(c.currentInstructions(), b...)
return posNewIns
}
func (c *Compiler) replaceInstruction(pos int, inst []byte) {
copy(c.currentInstructions()[pos:], inst)
if c.trace != nil {
c.printTrace(fmt.Sprintf("REPLC %s",
FormatInstructions(c.scopes[c.scopeIndex].instructions[pos:], pos)[0]))
}
}
func (c *Compiler) changeOperand(opPos int, operand ...int) {
op := Opcode(c.currentInstructions()[opPos])
inst := MakeInstruction(op, operand...)
c.replaceInstruction(opPos, inst)
}
// optimizeFunc performs some code-level optimization for the current function instructions
// it removes unreachable (dead code) instructions and adds "returns" instruction if needed.
func (c *Compiler) optimizeFunc(node ast.Node) {
// any instructions between RETURN and the function end
// or instructions between RETURN and jump target position
// are considered as unreachable.
// pass 1. identify all jump destinations
dsts := make(map[int]bool)
iterateInstructions(c.scopes[c.scopeIndex].instructions, func(pos int, opcode Opcode, operands []int) bool {
switch opcode {
case OpJump, OpJumpFalsy, OpAndJump, OpOrJump:
dsts[operands[0]] = true
}
return true
})
var newInsts []byte
// pass 2. eliminate dead code
posMap := make(map[int]int) // old position to new position
var dstIdx int
var deadCode bool
iterateInstructions(c.scopes[c.scopeIndex].instructions, func(pos int, opcode Opcode, operands []int) bool {
switch {
case opcode == OpReturn:
if deadCode {
return true
}
deadCode = true
case dsts[pos]:
dstIdx++
deadCode = false
case deadCode:
return true
}
posMap[pos] = len(newInsts)
newInsts = append(newInsts, MakeInstruction(opcode, operands...)...)
return true
})
// pass 3. update jump positions
var lastOp Opcode
var appendReturn bool
endPos := len(c.scopes[c.scopeIndex].instructions)
iterateInstructions(newInsts, func(pos int, opcode Opcode, operands []int) bool {
switch opcode {
case OpJump, OpJumpFalsy, OpAndJump, OpOrJump:
newDst, ok := posMap[operands[0]]
if ok {
copy(newInsts[pos:], MakeInstruction(opcode, newDst))
} else if endPos == operands[0] {
// there's a jump instruction that jumps to the end of function
// compiler should append "return".
appendReturn = true
} else {
panic(fmt.Errorf("invalid jump position: %d", newDst))
}
}
lastOp = opcode
return true
})
if lastOp != OpReturn {
appendReturn = true
}
// pass 4. update source map
newSourceMap := make(map[int]source.Pos)
for pos, srcPos := range c.scopes[c.scopeIndex].sourceMap {
newPos, ok := posMap[pos]
if ok {
newSourceMap[newPos] = srcPos
}
}
c.scopes[c.scopeIndex].instructions = newInsts
c.scopes[c.scopeIndex].sourceMap = newSourceMap
// append "return"
if appendReturn {
c.emit(node, OpReturn, 0)
}
}
func (c *Compiler) emit(node ast.Node, opcode Opcode, operands ...int) int {
filePos := source.NoPos
if node != nil {
filePos = node.Pos()
}
inst := MakeInstruction(opcode, operands...)
pos := c.addInstruction(inst)
c.scopes[c.scopeIndex].sourceMap[pos] = filePos
if c.trace != nil {
c.printTrace(fmt.Sprintf("EMIT %s",
FormatInstructions(c.scopes[c.scopeIndex].instructions[pos:], pos)[0]))
}
return pos
}
func (c *Compiler) printTrace(a ...interface{}) {
const (
dots = ". . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . "
n = len(dots)
)
i := 2 * c.indent
for i > n {
_, _ = fmt.Fprint(c.trace, dots)
i -= n
}
_, _ = fmt.Fprint(c.trace, dots[0:i])
_, _ = fmt.Fprintln(c.trace, a...)
}
func trace(c *Compiler, msg string) *Compiler {
c.printTrace(msg, "{")
c.indent++
return c
}
func un(c *Compiler) {
c.indent--
c.printTrace("}")
}

133
compiler/compiler_assign.go
View file

@ -1,133 +0,0 @@
package compiler
import (
"fmt"
"github.com/d5/tengo/compiler/ast"
"github.com/d5/tengo/compiler/token"
)
func (c *Compiler) compileAssign(node ast.Node, lhs, rhs []ast.Expr, op token.Token) error {
numLHS, numRHS := len(lhs), len(rhs)
if numLHS > 1 || numRHS > 1 {
return c.errorf(node, "tuple assignment not allowed")
}
// resolve and compile left-hand side
ident, selectors := resolveAssignLHS(lhs[0])
numSel := len(selectors)
if op == token.Define && numSel > 0 {
// using selector on new variable does not make sense
return c.errorf(node, "operator ':=' not allowed with selector")
}
symbol, depth, exists := c.symbolTable.Resolve(ident)
if op == token.Define {
if depth == 0 && exists {
return c.errorf(node, "'%s' redeclared in this block", ident)
}
symbol = c.symbolTable.Define(ident)
} else {
if !exists {
return c.errorf(node, "unresolved reference '%s'", ident)
}
}
// +=, -=, *=, /=
if op != token.Assign && op != token.Define {
if err := c.Compile(lhs[0]); err != nil {
return err
}
}
// compile RHSs
for _, expr := range rhs {
if err := c.Compile(expr); err != nil {
return err
}
}
switch op {
case token.AddAssign:
c.emit(node, OpBinaryOp, int(token.Add))
case token.SubAssign:
c.emit(node, OpBinaryOp, int(token.Sub))
case token.MulAssign:
c.emit(node, OpBinaryOp, int(token.Mul))
case token.QuoAssign:
c.emit(node, OpBinaryOp, int(token.Quo))
case token.RemAssign:
c.emit(node, OpBinaryOp, int(token.Rem))
case token.AndAssign:
c.emit(node, OpBinaryOp, int(token.And))
case token.OrAssign:
c.emit(node, OpBinaryOp, int(token.Or))
case token.AndNotAssign:
c.emit(node, OpBinaryOp, int(token.AndNot))
case token.XorAssign:
c.emit(node, OpBinaryOp, int(token.Xor))
case token.ShlAssign:
c.emit(node, OpBinaryOp, int(token.Shl))
case token.ShrAssign:
c.emit(node, OpBinaryOp, int(token.Shr))
}
// compile selector expressions (right to left)
for i := numSel - 1; i >= 0; i-- {
if err := c.Compile(selectors[i]); err != nil {
return err
}
}
switch symbol.Scope {
case ScopeGlobal:
if numSel > 0 {
c.emit(node, OpSetSelGlobal, symbol.Index, numSel)
} else {
c.emit(node, OpSetGlobal, symbol.Index)
}
case ScopeLocal:
if numSel > 0 {
c.emit(node, OpSetSelLocal, symbol.Index, numSel)
} else {
if op == token.Define && !symbol.LocalAssigned {
c.emit(node, OpDefineLocal, symbol.Index)
} else {
c.emit(node, OpSetLocal, symbol.Index)
}
}
// mark the symbol as local-assigned
symbol.LocalAssigned = true
case ScopeFree:
if numSel > 0 {
c.emit(node, OpSetSelFree, symbol.Index, numSel)
} else {
c.emit(node, OpSetFree, symbol.Index)
}
default:
panic(fmt.Errorf("invalid assignment variable scope: %s", symbol.Scope))
}
return nil
}
func resolveAssignLHS(expr ast.Expr) (name string, selectors []ast.Expr) {
switch term := expr.(type) {
case *ast.SelectorExpr:
name, selectors = resolveAssignLHS(term.Expr)
selectors = append(selectors, term.Sel)
return
case *ast.IndexExpr:
name, selectors = resolveAssignLHS(term.Expr)
selectors = append(selectors, term.Index)
case *ast.Ident:
name = term.Name
}
return
}

17
compiler/compiler_error_report_test.go
View file

@ -1,17 +0,0 @@
package compiler_test
import "testing"
func TestCompilerErrorReport(t *testing.T) {
expectError(t, `import("user1")`, "Compile Error: module 'user1' not found\n\tat test:1:1")
expectError(t, `a = 1`, "Compile Error: unresolved reference 'a'\n\tat test:1:1")
expectError(t, `a, b := 1, 2`, "Compile Error: tuple assignment not allowed\n\tat test:1:1")
expectError(t, `a.b := 1`, "not allowed with selector")
expectError(t, `a:=1; a:=3`, "Compile Error: 'a' redeclared in this block\n\tat test:1:7")
expectError(t, `return 5`, "Compile Error: return not allowed outside function\n\tat test:1:1")
expectError(t, `func() { break }`, "Compile Error: break not allowed outside loop\n\tat test:1:10")
expectError(t, `func() { continue }`, "Compile Error: continue not allowed outside loop\n\tat test:1:10")
expectError(t, `func() { export 5 }`, "Compile Error: export not allowed inside function\n\tat test:1:10")
}

181
compiler/compiler_for.go
View file

@ -1,181 +0,0 @@
package compiler
import (
"github.com/d5/tengo/compiler/ast"
)
func (c *Compiler) compileForStmt(stmt *ast.ForStmt) error {
c.symbolTable = c.symbolTable.Fork(true)
defer func() {
c.symbolTable = c.symbolTable.Parent(false)
}()
// init statement
if stmt.Init != nil {
if err := c.Compile(stmt.Init); err != nil {
return err
}
}
// pre-condition position
preCondPos := len(c.currentInstructions())
// condition expression
postCondPos := -1
if stmt.Cond != nil {
if err := c.Compile(stmt.Cond); err != nil {
return err
}
// condition jump position
postCondPos = c.emit(stmt, OpJumpFalsy, 0)
}
// enter loop
loop := c.enterLoop()
// body statement
if err := c.Compile(stmt.Body); err != nil {
c.leaveLoop()
return err
}
c.leaveLoop()
// post-body position
postBodyPos := len(c.currentInstructions())
// post statement
if stmt.Post != nil {
if err := c.Compile(stmt.Post); err != nil {
return err
}
}
// back to condition
c.emit(stmt, OpJump, preCondPos)
// post-statement position
postStmtPos := len(c.currentInstructions())
if postCondPos >= 0 {
c.changeOperand(postCondPos, postStmtPos)
}
// update all break/continue jump positions
for _, pos := range loop.Breaks {
c.changeOperand(pos, postStmtPos)
}
for _, pos := range loop.Continues {
c.changeOperand(pos, postBodyPos)
}
return nil
}
func (c *Compiler) compileForInStmt(stmt *ast.ForInStmt) error {
c.symbolTable = c.symbolTable.Fork(true)
defer func() {
c.symbolTable = c.symbolTable.Parent(false)
}()
// for-in statement is compiled like following:
//
// for :it := iterator(iterable); :it.next(); {
// k, v := :it.get() // DEFINE operator
//
// ... body ...
// }
//
// ":it" is a local variable but will be conflict with other user variables
// because character ":" is not allowed.
// init
// :it = iterator(iterable)
itSymbol := c.symbolTable.Define(":it")
if err := c.Compile(stmt.Iterable); err != nil {
return err
}
c.emit(stmt, OpIteratorInit)
if itSymbol.Scope == ScopeGlobal {
c.emit(stmt, OpSetGlobal, itSymbol.Index)
} else {
c.emit(stmt, OpDefineLocal, itSymbol.Index)
}
// pre-condition position
preCondPos := len(c.currentInstructions())
// condition
// :it.HasMore()
if itSymbol.Scope == ScopeGlobal {
c.emit(stmt, OpGetGlobal, itSymbol.Index)
} else {
c.emit(stmt, OpGetLocal, itSymbol.Index)
}
c.emit(stmt, OpIteratorNext)
// condition jump position
postCondPos := c.emit(stmt, OpJumpFalsy, 0)
// enter loop
loop := c.enterLoop()
// assign key variable
if stmt.Key.Name != "_" {
keySymbol := c.symbolTable.Define(stmt.Key.Name)
if itSymbol.Scope == ScopeGlobal {
c.emit(stmt, OpGetGlobal, itSymbol.Index)
} else {
c.emit(stmt, OpGetLocal, itSymbol.Index)
}
c.emit(stmt, OpIteratorKey)
if keySymbol.Scope == ScopeGlobal {
c.emit(stmt, OpSetGlobal, keySymbol.Index)
} else {
c.emit(stmt, OpDefineLocal, keySymbol.Index)
}
}
// assign value variable
if stmt.Value.Name != "_" {
valueSymbol := c.symbolTable.Define(stmt.Value.Name)
if itSymbol.Scope == ScopeGlobal {
c.emit(stmt, OpGetGlobal, itSymbol.Index)
} else {
c.emit(stmt, OpGetLocal, itSymbol.Index)
}
c.emit(stmt, OpIteratorValue)
if valueSymbol.Scope == ScopeGlobal {
c.emit(stmt, OpSetGlobal, valueSymbol.Index)
} else {
c.emit(stmt, OpDefineLocal, valueSymbol.Index)
}
}
// body statement
if err := c.Compile(stmt.Body); err != nil {
c.leaveLoop()
return err
}
c.leaveLoop()
// post-body position
postBodyPos := len(c.currentInstructions())
// back to condition
c.emit(stmt, OpJump, preCondPos)
// post-statement position
postStmtPos := len(c.currentInstructions())
c.changeOperand(postCondPos, postStmtPos)
// update all break/continue jump positions
for _, pos := range loop.Breaks {
c.changeOperand(pos, postStmtPos)
}
for _, pos := range loop.Continues {
c.changeOperand(pos, postBodyPos)
}
return nil
}

30
compiler/compiler_logical.go
View file

@ -1,30 +0,0 @@
package compiler
import (
"github.com/d5/tengo/compiler/ast"
"github.com/d5/tengo/compiler/token"
)
func (c *Compiler) compileLogical(node *ast.BinaryExpr) error {
// left side term
if err := c.Compile(node.LHS); err != nil {
return err
}
// jump position
var jumpPos int
if node.Token == token.LAnd {
jumpPos = c.emit(node, OpAndJump, 0)
} else {
jumpPos = c.emit(node, OpOrJump, 0)
}
// right side term
if err := c.Compile(node.RHS); err != nil {
return err
}
c.changeOperand(jumpPos, len(c.currentInstructions()))
return nil
}

31
compiler/compiler_loops.go
View file

@ -1,31 +0,0 @@
package compiler
func (c *Compiler) enterLoop() *Loop {
loop := &Loop{}
c.loops = append(c.loops, loop)
c.loopIndex++
if c.trace != nil {
c.printTrace("LOOPE", c.loopIndex)
}
return loop
}
func (c *Compiler) leaveLoop() {
if c.trace != nil {
c.printTrace("LOOPL", c.loopIndex)
}
c.loops = c.loops[:len(c.loops)-1]
c.loopIndex--
}
func (c *Compiler) currentLoop() *Loop {
if c.loopIndex >= 0 {
return c.loops[c.loopIndex]
}
return nil
}

79
compiler/compiler_module.go
View file

@ -1,79 +0,0 @@
package compiler
import (
"github.com/d5/tengo/compiler/ast"
"github.com/d5/tengo/compiler/parser"
"github.com/d5/tengo/objects"
)
func (c *Compiler) checkCyclicImports(node ast.Node, modulePath string) error {
if c.modulePath == modulePath {
return c.errorf(node, "cyclic module import: %s", modulePath)
} else if c.parent != nil {
return c.parent.checkCyclicImports(node, modulePath)
}
return nil
}
func (c *Compiler) compileModule(node ast.Node, moduleName, modulePath string, src []byte) (*objects.CompiledFunction, error) {
if err := c.checkCyclicImports(node, modulePath); err != nil {
return nil, err
}
compiledModule, exists := c.loadCompiledModule(modulePath)
if exists {
return compiledModule, nil
}
modFile := c.file.Set().AddFile(moduleName, -1, len(src))
p := parser.NewParser(modFile, src, nil)
file, err := p.ParseFile()
if err != nil {
return nil, err
}
symbolTable := NewSymbolTable()
// inherit builtin functions
for _, sym := range c.symbolTable.BuiltinSymbols() {
symbolTable.DefineBuiltin(sym.Index, sym.Name)
}
// no global scope for the module
symbolTable = symbolTable.Fork(false)
// compile module
moduleCompiler := c.fork(modFile, modulePath, symbolTable)
if err := moduleCompiler.Compile(file); err != nil {
return nil, err
}
// code optimization
moduleCompiler.optimizeFunc(node)
compiledFunc := moduleCompiler.Bytecode().MainFunction
compiledFunc.NumLocals = symbolTable.MaxSymbols()
c.storeCompiledModule(modulePath, compiledFunc)
return compiledFunc, nil
}
func (c *Compiler) loadCompiledModule(modulePath string) (mod *objects.CompiledFunction, ok bool) {
if c.parent != nil {
return c.parent.loadCompiledModule(modulePath)
}
mod, ok = c.compiledModules[modulePath]
return
}
func (c *Compiler) storeCompiledModule(modulePath string, module *objects.CompiledFunction) {
if c.parent != nil {
c.parent.storeCompiledModule(modulePath, module)
}
c.compiledModules[modulePath] = module
}

124
compiler/compiler_optimize_test.go
View file

@ -1,124 +0,0 @@
package compiler_test
import (
"testing"
"github.com/d5/tengo/compiler"
)
func TestCompilerDeadCode(t *testing.T) {
expect(t, `
func() {
a := 4
return a
b := 5 // dead code from here
c := a
return b
}`,
bytecode(
concat(
compiler.MakeInstruction(compiler.OpConstant, 2),
compiler.MakeInstruction(compiler.OpPop)),
objectsArray(
intObject(4),
intObject(5),
compiledFunction(0, 0,
compiler.MakeInstruction(compiler.OpConstant, 0),
compiler.MakeInstruction(compiler.OpDefineLocal, 0),
compiler.MakeInstruction(compiler.OpGetLocal, 0),
compiler.MakeInstruction(compiler.OpReturn, 1)))))
expect(t, `
func() {
if true {
return 5
a := 4 // dead code from here
b := a
return b
} else {
return 4
c := 5 // dead code from here
d := c
return d
}
}`, bytecode(
concat(
compiler.MakeInstruction(compiler.OpConstant, 2),
compiler.MakeInstruction(compiler.OpPop)),
objectsArray(
intObject(5),
intObject(4),
compiledFunction(0, 0,
compiler.MakeInstruction(compiler.OpTrue),
compiler.MakeInstruction(compiler.OpJumpFalsy, 9),
compiler.MakeInstruction(compiler.OpConstant, 0),
compiler.MakeInstruction(compiler.OpReturn, 1),
compiler.MakeInstruction(compiler.OpConstant, 1),
compiler.MakeInstruction(compiler.OpReturn, 1)))))
expect(t, `
func() {
a := 1
for {
if a == 5 {
return 10
}
5 + 5
return 20
b := a
return b
}
}`, bytecode(
concat(
compiler.MakeInstruction(compiler.OpConstant, 4),
compiler.MakeInstruction(compiler.OpPop)),
objectsArray(
intObject(1),
intObject(5),
intObject(10),
intObject(20),
compiledFunction(0, 0,
compiler.MakeInstruction(compiler.OpConstant, 0),
compiler.MakeInstruction(compiler.OpDefineLocal, 0),
compiler.MakeInstruction(compiler.OpGetLocal, 0),
compiler.MakeInstruction(compiler.OpConstant, 1),
compiler.MakeInstruction(compiler.OpEqual),
compiler.MakeInstruction(compiler.OpJumpFalsy, 19),
compiler.MakeInstruction(compiler.OpConstant, 2),
compiler.MakeInstruction(compiler.OpReturn, 1),
compiler.MakeInstruction(compiler.OpConstant, 1),
compiler.MakeInstruction(compiler.OpConstant, 1),
compiler.MakeInstruction(compiler.OpBinaryOp, 11),
compiler.MakeInstruction(compiler.OpPop),
compiler.MakeInstruction(compiler.OpConstant, 3),
compiler.MakeInstruction(compiler.OpReturn, 1)))))
expect(t, `
func() {
if true {
return 5
a := 4 // dead code from here
b := a
return b
} else {
return 4
c := 5 // dead code from here
d := c
return d
}
}`, bytecode(
concat(
compiler.MakeInstruction(compiler.OpConstant, 2),
compiler.MakeInstruction(compiler.OpPop)),
objectsArray(
intObject(5),
intObject(4),
compiledFunction(0, 0,
compiler.MakeInstruction(compiler.OpTrue),
compiler.MakeInstruction(compiler.OpJumpFalsy, 9),
compiler.MakeInstruction(compiler.OpConstant, 0),
compiler.MakeInstruction(compiler.OpReturn, 1),
compiler.MakeInstruction(compiler.OpConstant, 1),
compiler.MakeInstruction(compiler.OpReturn, 1)))))
}

43
compiler/compiler_scopes.go
View file

@ -1,43 +0,0 @@
package compiler
import "github.com/d5/tengo/compiler/source"
func (c *Compiler) currentInstructions() []byte {
return c.scopes[c.scopeIndex].instructions
}
func (c *Compiler) currentSourceMap() map[int]source.Pos {
return c.scopes[c.scopeIndex].sourceMap
}
func (c *Compiler) enterScope() {
scope := CompilationScope{
symbolInit: make(map[string]bool),
sourceMap: make(map[int]source.Pos),
}
c.scopes = append(c.scopes, scope)
c.scopeIndex++
c.symbolTable = c.symbolTable.Fork(false)
if c.trace != nil {
c.printTrace("SCOPE", c.scopeIndex)
}
}
func (c *Compiler) leaveScope() (instructions []byte, sourceMap map[int]source.Pos) {
instructions = c.currentInstructions()
sourceMap = c.currentSourceMap()
c.scopes = c.scopes[:len(c.scopes)-1]
c.scopeIndex--
c.symbolTable = c.symbolTable.Parent(true)
if c.trace != nil {
c.printTrace("SCOPL", c.scopeIndex)
}
return
}

69
compiler/compiler_scopes_test.go
View file

@ -1,69 +0,0 @@
package compiler_test
import (
"testing"
"github.com/d5/tengo/compiler"
)
func TestCompilerScopes(t *testing.T) {
expect(t, `
if a := 1; a {
a = 2
b := a
} else {
a = 3
b := a
}`, bytecode(
concat(
compiler.MakeInstruction(compiler.OpConstant, 0),
compiler.MakeInstruction(compiler.OpSetGlobal, 0),
compiler.MakeInstruction(compiler.OpGetGlobal, 0),
compiler.MakeInstruction(compiler.OpJumpFalsy, 27),
compiler.MakeInstruction(compiler.OpConstant, 1),
compiler.MakeInstruction(compiler.OpSetGlobal, 0),
compiler.MakeInstruction(compiler.OpGetGlobal, 0),
compiler.MakeInstruction(compiler.OpSetGlobal, 1),
compiler.MakeInstruction(compiler.OpJump, 39),
compiler.MakeInstruction(compiler.OpConstant, 2),
compiler.MakeInstruction(compiler.OpSetGlobal, 0),
compiler.MakeInstruction(compiler.OpGetGlobal, 0),
compiler.MakeInstruction(compiler.OpSetGlobal, 1)),
objectsArray(
intObject(1),
intObject(2),
intObject(3))))
expect(t, `
func() {
if a := 1; a {
a = 2
b := a
} else {
a = 3
b := a
}
}`, bytecode(
concat(
compiler.MakeInstruction(compiler.OpConstant, 3),
compiler.MakeInstruction(compiler.OpPop)),
objectsArray(
intObject(1),
intObject(2),
intObject(3),
compiledFunction(0, 0,
compiler.MakeInstruction(compiler.OpConstant, 0),
compiler.MakeInstruction(compiler.OpDefineLocal, 0),
compiler.MakeInstruction(compiler.OpGetLocal, 0),
compiler.MakeInstruction(compiler.OpJumpFalsy, 22),
compiler.MakeInstruction(compiler.OpConstant, 1),
compiler.MakeInstruction(compiler.OpSetLocal, 0),
compiler.MakeInstruction(compiler.OpGetLocal, 0),
compiler.MakeInstruction(compiler.OpDefineLocal, 1),
compiler.MakeInstruction(compiler.OpJump, 31),
compiler.MakeInstruction(compiler.OpConstant, 2),
compiler.MakeInstruction(compiler.OpSetLocal, 0),
compiler.MakeInstruction(compiler.OpGetLocal, 0),
compiler.MakeInstruction(compiler.OpDefineLocal, 1),
compiler.MakeInstruction(compiler.OpReturn, 0)))))
}

1068
compiler/compiler_test.go
View file

File diff suppressed because it is too large Load diff

8
compiler/emitted_instruction.go
View file

@ -1,8 +0,0 @@
package compiler
// EmittedInstruction represents an opcode
// with its emitted position.
type EmittedInstruction struct {
Opcode Opcode
Position int
}

20
compiler/error.go
View file

@ -1,20 +0,0 @@
package compiler
import (
"fmt"
"github.com/d5/tengo/compiler/ast"
"github.com/d5/tengo/compiler/source"
)
// Error represents a compiler error.
type Error struct {
fileSet *source.FileSet
node ast.Node
error error
}
func (e *Error) Error() string {
filePos := e.fileSet.Position(e.node.Pos())
return fmt.Sprintf("Compile Error: %s\n\tat %s", e.error.Error(), filePos)
}

65
compiler/instructions_test.go
View file

@ -1,65 +0,0 @@
package compiler_test
import (
"strings"
"testing"
"github.com/d5/tengo/assert"
"github.com/d5/tengo/compiler"
)
func TestInstructions_String(t *testing.T) {
assertInstructionString(t,
[][]byte{
compiler.MakeInstruction(compiler.OpConstant, 1),
compiler.MakeInstruction(compiler.OpConstant, 2),
compiler.MakeInstruction(compiler.OpConstant, 65535),
},
`0000 CONST 1
0003 CONST 2
0006 CONST 65535`)
assertInstructionString(t,
[][]byte{
compiler.MakeInstruction(compiler.OpBinaryOp, 11),
compiler.MakeInstruction(compiler.OpConstant, 2),
compiler.MakeInstruction(compiler.OpConstant, 65535),
},
`0000 BINARYOP 11
0002 CONST 2
0005 CONST 65535`)
assertInstructionString(t,
[][]byte{
compiler.MakeInstruction(compiler.OpBinaryOp, 11),
compiler.MakeInstruction(compiler.OpGetLocal, 1),
compiler.MakeInstruction(compiler.OpConstant, 2),
compiler.MakeInstruction(compiler.OpConstant, 65535),
},
`0000 BINARYOP 11
0002 GETL 1
0004 CONST 2
0007 CONST 65535`)
}
func TestMakeInstruction(t *testing.T) {
makeInstruction(t, []byte{byte(compiler.OpConstant), 0, 0}, compiler.OpConstant, 0)
makeInstruction(t, []byte{byte(compiler.OpConstant), 0, 1}, compiler.OpConstant, 1)
makeInstruction(t, []byte{byte(compiler.OpConstant), 255, 254}, compiler.OpConstant, 65534)
makeInstruction(t, []byte{byte(compiler.OpPop)}, compiler.OpPop)
makeInstruction(t, []byte{byte(compiler.OpTrue)}, compiler.OpTrue)
makeInstruction(t, []byte{byte(compiler.OpFalse)}, compiler.OpFalse)
}
func assertInstructionString(t *testing.T, instructions [][]byte, expected string) {
concatted := make([]byte, 0)
for _, e := range instructions {
concatted = append(concatted, e...)
}
assert.Equal(t, expected, strings.Join(compiler.FormatInstructions(concatted, 0), "\n"))
}
func makeInstruction(t *testing.T, expected []byte, opcode compiler.Opcode, operands ...int) {
inst := compiler.MakeInstruction(opcode, operands...)
assert.Equal(t, expected, []byte(inst))
}

8
compiler/loop.go
View file

@ -1,8 +0,0 @@
package compiler
// Loop represents a loop construct that
// the compiler uses to track the current loop.
type Loop struct {
Continues []int
Breaks []int
}

4
compiler/module_loader.go
View file

@ -1,4 +0,0 @@
package compiler
// ModuleLoader should take a module name and return the module data.
type ModuleLoader func(moduleName string) ([]byte, error)

20
compiler/opcodes_test.go
View file

@ -1,20 +0,0 @@
package compiler_test
import (
"testing"
"github.com/d5/tengo/assert"
"github.com/d5/tengo/compiler"
)
func TestReadOperands(t *testing.T) {
assertReadOperand(t, compiler.OpConstant, []int{65535}, 2)
}
func assertReadOperand(t *testing.T, opcode compiler.Opcode, operands []int, expectedBytes int) {
inst := compiler.MakeInstruction(opcode, operands...)
numOperands := compiler.OpcodeOperands[opcode]
operandsRead, read := compiler.ReadOperands(numOperands, inst[1:])
assert.Equal(t, expectedBytes, read)
assert.Equal(t, operands, operandsRead)
}

21
compiler/parser/error.go
View file

@ -1,21 +0,0 @@
package parser
import (
"fmt"
"github.com/d5/tengo/compiler/source"
)
// Error represents a parser error.
type Error struct {
Pos source.FilePos
Msg string
}
func (e Error) Error() string {
if e.Pos.Filename != "" || e.Pos.IsValid() {
return fmt.Sprintf("Parse Error: %s\n\tat %s", e.Msg, e.Pos)
}
return fmt.Sprintf("Parse Error: %s", e.Msg)
}

68
compiler/parser/error_list.go
View file

@ -1,68 +0,0 @@
package parser
import (
"fmt"
"sort"
"github.com/d5/tengo/compiler/source"
)
// ErrorList is a collection of parser errors.
type ErrorList []*Error
// Add adds a new parser error to the collection.
func (p *ErrorList) Add(pos source.FilePos, msg string) {
*p = append(*p, &Error{pos, msg})
}
// Len returns the number of elements in the collection.
func (p ErrorList) Len() int {
return len(p)
}
func (p ErrorList) Swap(i, j int) {
p[i], p[j] = p[j], p[i]
}
func (p ErrorList) Less(i, j int) bool {
e := &p[i].Pos
f := &p[j].Pos
if e.Filename != f.Filename {
return e.Filename < f.Filename
}
if e.Line != f.Line {
return e.Line < f.Line
}
if e.Column != f.Column {
return e.Column < f.Column
}
return p[i].Msg < p[j].Msg
}
// Sort sorts the collection.
func (p ErrorList) Sort() {
sort.Sort(p)
}
func (p ErrorList) Error() string {
switch len(p) {
case 0:
return "no errors"
case 1:
return p[0].Error()
}
return fmt.Sprintf("%s (and %d more errors)", p[0], len(p)-1)
}
// Err returns an error.
func (p ErrorList) Err() error {
if len(p) == 0 {
return nil
}
return p
}

18
compiler/parser/error_list_test.go
View file

@ -1,18 +0,0 @@
package parser_test
import (
"testing"
"github.com/d5/tengo/assert"
"github.com/d5/tengo/compiler/parser"
"github.com/d5/tengo/compiler/source"
)
func TestErrorList_Sort(t *testing.T) {
var list parser.ErrorList
list.Add(source.FilePos{Offset: 20, Line: 2, Column: 10}, "error 2")
list.Add(source.FilePos{Offset: 30, Line: 3, Column: 10}, "error 3")
list.Add(source.FilePos{Offset: 10, Line: 1, Column: 10}, "error 1")
list.Sort()
assert.Equal(t, "Parse Error: error 1\n\tat 1:10 (and 2 more errors)", list.Error())
}

14
compiler/parser/error_test.go
View file

@ -1,14 +0,0 @@
package parser_test
import (
"testing"
"github.com/d5/tengo/assert"
"github.com/d5/tengo/compiler/parser"
"github.com/d5/tengo/compiler/source"
)
func TestError_Error(t *testing.T) {
err := &parser.Error{Pos: source.FilePos{Offset: 10, Line: 1, Column: 10}, Msg: "test"}
assert.Equal(t, "Parse Error: test\n\tat 1:10", err.Error())
}

17
compiler/parser/parse_source.go
View file

@ -1,17 +0,0 @@
package parser
import (
"io"
"github.com/d5/tengo/compiler/ast"
"github.com/d5/tengo/compiler/source"
)
// ParseSource parses source code 'src' and builds an AST.
func ParseSource(filename string, src []byte, trace io.Writer) (res *ast.File, err error) {
fileSet := source.NewFileSet()
file := fileSet.AddFile(filename, -1, len(src))
p := NewParser(file, src, trace)
return p.ParseFile()
}

106
compiler/parser/parser_array_test.go
View file

@ -1,106 +0,0 @@
package parser_test
import (
"testing"
"github.com/d5/tengo/compiler/ast"
"github.com/d5/tengo/compiler/token"
)
func TestArray(t *testing.T) {
expect(t, "[1, 2, 3]", func(p pfn) []ast.Stmt {
return stmts(
exprStmt(
arrayLit(p(1, 1), p(1, 9),
intLit(1, p(1, 2)),
intLit(2, p(1, 5)),
intLit(3, p(1, 8)))))
})
expect(t, `
[
1,
2,
3
]`, func(p pfn) []ast.Stmt {
return stmts(
exprStmt(
arrayLit(p(2, 1), p(6, 1),
intLit(1, p(3, 2)),
intLit(2, p(4, 2)),
intLit(3, p(5, 2)))))
})
expect(t, `
[
1,
2,
3
]`, func(p pfn) []ast.Stmt {
return stmts(
exprStmt(
arrayLit(p(2, 1), p(7, 1),
intLit(1, p(3, 2)),
intLit(2, p(4, 2)),
intLit(3, p(5, 2)))))
})
expect(t, `[1, "foo", 12.34]`, func(p pfn) []ast.Stmt {
return stmts(
exprStmt(
arrayLit(p(1, 1), p(1, 17),
intLit(1, p(1, 2)),
stringLit("foo", p(1, 5)),
floatLit(12.34, p(1, 12)))))
})
expect(t, "a = [1, 2, 3]", func(p pfn) []ast.Stmt {
return stmts(
assignStmt(
exprs(ident("a", p(1, 1))),
exprs(arrayLit(p(1, 5), p(1, 13),
intLit(1, p(1, 6)),
intLit(2, p(1, 9)),
intLit(3, p(1, 12)))),
token.Assign,
p(1, 3)))
})
expect(t, "a = [1 + 2, b * 4, [4, c]]", func(p pfn) []ast.Stmt {
return stmts(
assignStmt(
exprs(ident("a", p(1, 1))),
exprs(arrayLit(p(1, 5), p(1, 26),
binaryExpr(
intLit(1, p(1, 6)),
intLit(2, p(1, 10)),
token.Add,
p(1, 8)),
binaryExpr(
ident("b", p(1, 13)),
intLit(4, p(1, 17)),
token.Mul,
p(1, 15)),
arrayLit(p(1, 20), p(1, 25),
intLit(4, p(1, 21)),
ident("c", p(1, 24))))),
token.Assign,
p(1, 3)))
})
expectError(t, `[1, 2, 3,]`)
expectError(t, `
[
1,
2,
3,
]`)
expectError(t, `
[
1,
2,
3,
]`)
expectError(t, `[1, 2, 3, ,]`)
}

132
compiler/parser/parser_assignment_test.go
View file

@ -1,132 +0,0 @@
package parser_test
import (
"testing"
"github.com/d5/tengo/compiler/ast"
"github.com/d5/tengo/compiler/token"
)
func TestAssignment(t *testing.T) {
expect(t, "a = 5", func(p pfn) []ast.Stmt {
return stmts(
assignStmt(
exprs(ident("a", p(1, 1))),
exprs(intLit(5, p(1, 5))),
token.Assign,
p(1, 3)))
})
expect(t, "a := 5", func(p pfn) []ast.Stmt {
return stmts(
assignStmt(
exprs(ident("a", p(1, 1))),
exprs(intLit(5, p(1, 6))),
token.Define,
p(1, 3)))
})
expect(t, "a, b = 5, 10", func(p pfn) []ast.Stmt {
return stmts(
assignStmt(
exprs(
ident("a", p(1, 1)),
ident("b", p(1, 4))),
exprs(
intLit(5, p(1, 8)),
intLit(10, p(1, 11))),
token.Assign,
p(1, 6)))
})
expect(t, "a, b := 5, 10", func(p pfn) []ast.Stmt {
return stmts(
assignStmt(
exprs(
ident("a", p(1, 1)),
ident("b", p(1, 4))),
exprs(
intLit(5, p(1, 9)),
intLit(10, p(1, 12))),
token.Define,
p(1, 6)))
})
expect(t, "a, b = a + 2, b - 8", func(p pfn) []ast.Stmt {
return stmts(
assignStmt(
exprs(
ident("a", p(1, 1)),
ident("b", p(1, 4))),
exprs(
binaryExpr(
ident("a", p(1, 8)),
intLit(2, p(1, 12)),
token.Add,
p(1, 10)),
binaryExpr(
ident("b", p(1, 15)),
intLit(8, p(1, 19)),
token.Sub,
p(1, 17))),
token.Assign,
p(1, 6)))
})
expect(t, "a = [1, 2, 3]", func(p pfn) []ast.Stmt {
return stmts(
assignStmt(
exprs(ident("a", p(1, 1))),
exprs(arrayLit(p(1, 5), p(1, 13),
intLit(1, p(1, 6)),
intLit(2, p(1, 9)),
intLit(3, p(1, 12)))),
token.Assign,
p(1, 3)))
})
expect(t, "a = [1 + 2, b * 4, [4, c]]", func(p pfn) []ast.Stmt {
return stmts(
assignStmt(
exprs(ident("a", p(1, 1))),
exprs(arrayLit(p(1, 5), p(1, 26),
binaryExpr(
intLit(1, p(1, 6)),
intLit(2, p(1, 10)),
token.Add,
p(1, 8)),
binaryExpr(
ident("b", p(1, 13)),
intLit(4, p(1, 17)),
token.Mul,
p(1, 15)),
arrayLit(p(1, 20), p(1, 25),
intLit(4, p(1, 21)),
ident("c", p(1, 24))))),
token.Assign,
p(1, 3)))
})
expect(t, "a += 5", func(p pfn) []ast.Stmt {
return stmts(
assignStmt(
exprs(ident("a", p(1, 1))),
exprs(intLit(5, p(1, 6))),
token.AddAssign,
p(1, 3)))
})
expect(t, "a *= 5 + 10", func(p pfn) []ast.Stmt {
return stmts(
assignStmt(
exprs(ident("a", p(1, 1))),
exprs(
binaryExpr(
intLit(5, p(1, 6)),
intLit(10, p(1, 10)),
token.Add,
p(1, 8))),
token.MulAssign,
p(1, 3)))
})
}

41
compiler/parser/parser_boolean_test.go
View file

@ -1,41 +0,0 @@
package parser_test
import (
"testing"
"github.com/d5/tengo/compiler/ast"
"github.com/d5/tengo/compiler/token"
)
func TestBoolean(t *testing.T) {
expect(t, "true", func(p pfn) []ast.Stmt {
return stmts(
exprStmt(
boolLit(true, p(1, 1))))
})
expect(t, "false", func(p pfn) []ast.Stmt {
return stmts(
exprStmt(
boolLit(false, p(1, 1))))
})
expect(t, "true != false", func(p pfn) []ast.Stmt {
return stmts(
exprStmt(
binaryExpr(
boolLit(true, p(1, 1)),
boolLit(false, p(1, 9)),
token.NotEqual,
p(1, 6))))
})
expect(t, "!false", func(p pfn) []ast.Stmt {
return stmts(
exprStmt(
unaryExpr(
boolLit(false, p(1, 2)),
token.Not,
p(1, 1))))
})
}

144
compiler/parser/parser_call_test.go
View file

@ -1,144 +0,0 @@
package parser_test
import (
"testing"
"github.com/d5/tengo/compiler/ast"
"github.com/d5/tengo/compiler/token"
)
func TestCall(t *testing.T) {
expect(t, "add(1, 2, 3)", func(p pfn) []ast.Stmt {
return stmts(
exprStmt(
callExpr(
ident("add", p(1, 1)),
p(1, 4), p(1, 12),
intLit(1, p(1, 5)),
intLit(2, p(1, 8)),
intLit(3, p(1, 11)))))
})
expect(t, "a = add(1, 2, 3)", func(p pfn) []ast.Stmt {
return stmts(
assignStmt(
exprs(
ident("a", p(1, 1))),
exprs(
callExpr(
ident("add", p(1, 5)),
p(1, 8), p(1, 16),
intLit(1, p(1, 9)),
intLit(2, p(1, 12)),
intLit(3, p(1, 15)))),
token.Assign,
p(1, 3)))
})
expect(t, "a, b = add(1, 2, 3)", func(p pfn) []ast.Stmt {
return stmts(
assignStmt(
exprs(
ident("a", p(1, 1)),
ident("b", p(1, 4))),
exprs(
callExpr(
ident("add", p(1, 8)),
p(1, 11), p(1, 19),
intLit(1, p(1, 12)),
intLit(2, p(1, 15)),
intLit(3, p(1, 18)))),
token.Assign,
p(1, 6)))
})
expect(t, "add(a + 1, 2 * 1, (b + c))", func(p pfn) []ast.Stmt {
return stmts(
exprStmt(
callExpr(
ident("add", p(1, 1)),
p(1, 4), p(1, 26),
binaryExpr(
ident("a", p(1, 5)),
intLit(1, p(1, 9)),
token.Add,
p(1, 7)),
binaryExpr(
intLit(2, p(1, 12)),
intLit(1, p(1, 16)),
token.Mul,
p(1, 14)),
parenExpr(
binaryExpr(
ident("b", p(1, 20)),
ident("c", p(1, 24)),
token.Add,
p(1, 22)),
p(1, 19), p(1, 25)))))
})
expectString(t, "a + add(b * c) + d", "((a + add((b * c))) + d)")
expectString(t, "add(a, b, 1, 2 * 3, 4 + 5, add(6, 7 * 8))", "add(a, b, 1, (2 * 3), (4 + 5), add(6, (7 * 8)))")
expectString(t, "f1(a) + f2(b) * f3(c)", "(f1(a) + (f2(b) * f3(c)))")
expectString(t, "(f1(a) + f2(b)) * f3(c)", "(((f1(a) + f2(b))) * f3(c))")
expect(t, "func(a, b) { a + b }(1, 2)", func(p pfn) []ast.Stmt {
return stmts(
exprStmt(
callExpr(
funcLit(
funcType(
identList(
p(1, 5), p(1, 10),
false,
ident("a", p(1, 6)),
ident("b", p(1, 9))),
p(1, 1)),
blockStmt(
p(1, 12), p(1, 20),
exprStmt(
binaryExpr(
ident("a", p(1, 14)),
ident("b", p(1, 18)),
token.Add,
p(1, 16))))),
p(1, 21), p(1, 26),
intLit(1, p(1, 22)),
intLit(2, p(1, 25)))))
})
expect(t, `a.b()`, func(p pfn) []ast.Stmt {
return stmts(
exprStmt(
callExpr(
selectorExpr(
ident("a", p(1, 1)),
stringLit("b", p(1, 3))),
p(1, 4), p(1, 5))))
})
expect(t, `a.b.c()`, func(p pfn) []ast.Stmt {
return stmts(
exprStmt(
callExpr(
selectorExpr(
selectorExpr(
ident("a", p(1, 1)),
stringLit("b", p(1, 3))),
stringLit("c", p(1, 5))),
p(1, 6), p(1, 7))))
})
expect(t, `a["b"].c()`, func(p pfn) []ast.Stmt {
return stmts(
exprStmt(
callExpr(
selectorExpr(
indexExpr(
ident("a", p(1, 1)),
stringLit("b", p(1, 3)),
p(1, 2), p(1, 6)),
stringLit("c", p(1, 8))),
p(1, 9), p(1, 10))))
})
}

24
compiler/parser/parser_char_test.go
View file

@ -1,24 +0,0 @@
package parser_test
import (
"testing"
"github.com/d5/tengo/compiler/ast"
)
func TestChar(t *testing.T) {
expect(t, `'A'`, func(p pfn) []ast.Stmt {
return stmts(
exprStmt(
charLit('A', 1)))
})
expect(t, `'九'`, func(p pfn) []ast.Stmt {
return stmts(
exprStmt(
charLit('九', 1)))
})
expectError(t, `''`)
expectError(t, `'AB'`)
expectError(t, `'A九'`)
}

48
compiler/parser/parser_cond_test.go
View file

@ -1,48 +0,0 @@
package parser_test
import (
"testing"
"github.com/d5/tengo/compiler/ast"
)
func TestCondExpr(t *testing.T) {
expect(t, "a ? b : c", func(p pfn) []ast.Stmt {
return stmts(
exprStmt(
condExpr(
ident("a", p(1, 1)),
ident("b", p(1, 5)),
ident("c", p(1, 9)),
p(1, 3),
p(1, 7))))
})
expect(t, `a ?
b :
c`, func(p pfn) []ast.Stmt {
return stmts(
exprStmt(
condExpr(
ident("a", p(1, 1)),
ident("b", p(1, 5)),
ident("c", p(1, 9)),
p(1, 3),
p(1, 7))))
})
expectString(t, `a ? b : c`, "(a ? b : c)")
expectString(t, `a + b ? c - d : e * f`, "((a + b) ? (c - d) : (e * f))")
expectString(t, `a == b ? c + (d / e) : f ? g : h + i`, "((a == b) ? (c + ((d / e))) : (f ? g : (h + i)))")
expectString(t, `(a + b) ? (c - d) : (e * f)`, "(((a + b)) ? ((c - d)) : ((e * f)))")
expectString(t, `a + (b ? c : d) - e`, "((a + ((b ? c : d))) - e)")
expectString(t, `a ? b ? c : d : e`, "(a ? (b ? c : d) : e)")
expectString(t, `a := b ? c : d`, "a := (b ? c : d)")
expectString(t, `x := a ? b ? c : d : e`, "x := (a ? (b ? c : d) : e)")
// ? : should be at the end of each line if it's multi-line
expectError(t, `a
? b
: c`)
expectError(t, `a ? (b : e)`)
expectError(t, `(a ? b) : e`)
}

43
compiler/parser/parser_error_test.go
View file

@ -1,43 +0,0 @@
package parser_test
import (
"testing"
"github.com/d5/tengo/compiler/ast"
"github.com/d5/tengo/compiler/token"
)
func TestError(t *testing.T) {
expect(t, `error(1234)`, func(p pfn) []ast.Stmt {
return stmts(
exprStmt(
errorExpr(p(1, 1), intLit(1234, p(1, 7)), p(1, 6), p(1, 11))))
})
expect(t, `err1 := error("some error")`, func(p pfn) []ast.Stmt {
return stmts(
assignStmt(
exprs(ident("err1", p(1, 1))),
exprs(errorExpr(p(1, 9), stringLit("some error", p(1, 15)), p(1, 14), p(1, 27))),
token.Define, p(1, 6)))
})
expect(t, `return error("some error")`, func(p pfn) []ast.Stmt {
return stmts(
returnStmt(p(1, 1),
errorExpr(p(1, 8), stringLit("some error", p(1, 14)), p(1, 13), p(1, 26))))
})
expect(t, `return error("some" + "error")`, func(p pfn) []ast.Stmt {
return stmts(
returnStmt(p(1, 1),
errorExpr(p(1, 8),
binaryExpr(
stringLit("some", p(1, 14)),
stringLit("error", p(1, 23)),
token.Add, p(1, 21)),
p(1, 13), p(1, 30))))
})
expectError(t, `error()`) // must have a value
}

66
compiler/parser/parser_for_in_test.go
View file

@ -1,66 +0,0 @@
package parser_test
import (
"testing"
"github.com/d5/tengo/compiler/ast"
)
func TestForIn(t *testing.T) {
expect(t, "for x in y {}", func(p pfn) []ast.Stmt {
return stmts(
forInStmt(
ident("_", p(1, 5)),
ident("x", p(1, 5)),
ident("y", p(1, 10)),
blockStmt(p(1, 12), p(1, 13)),
p(1, 1)))
})
expect(t, "for _ in y {}", func(p pfn) []ast.Stmt {
return stmts(
forInStmt(
ident("_", p(1, 5)),
ident("_", p(1, 5)),
ident("y", p(1, 10)),
blockStmt(p(1, 12), p(1, 13)),
p(1, 1)))
})
expect(t, "for x in [1, 2, 3] {}", func(p pfn) []ast.Stmt {
return stmts(
forInStmt(
ident("_", p(1, 5)),
ident("x", p(1, 5)),
arrayLit(
p(1, 10), p(1, 18),
intLit(1, p(1, 11)),
intLit(2, p(1, 14)),
intLit(3, p(1, 17))),
blockStmt(p(1, 20), p(1, 21)),
p(1, 1)))
})
expect(t, "for x, y in z {}", func(p pfn) []ast.Stmt {
return stmts(
forInStmt(
ident("x", p(1, 5)),
ident("y", p(1, 8)),
ident("z", p(1, 13)),
blockStmt(p(1, 15), p(1, 16)),
p(1, 1)))
})
expect(t, "for x, y in {k1: 1, k2: 2} {}", func(p pfn) []ast.Stmt {
return stmts(
forInStmt(
ident("x", p(1, 5)),
ident("y", p(1, 8)),
mapLit(
p(1, 13), p(1, 26),
mapElementLit("k1", p(1, 14), p(1, 16), intLit(1, p(1, 18))),
mapElementLit("k2", p(1, 21), p(1, 23), intLit(2, p(1, 25)))),
blockStmt(p(1, 28), p(1, 29)),
p(1, 1)))
})
}

118
compiler/parser/parser_for_test.go
View file

@ -1,118 +0,0 @@
package parser_test
import (
"testing"
"github.com/d5/tengo/compiler/ast"
"github.com/d5/tengo/compiler/token"
)
func TestFor(t *testing.T) {
expect(t, "for {}", func(p pfn) []ast.Stmt {
return stmts(
forStmt(nil, nil, nil, blockStmt(p(1, 5), p(1, 6)), p(1, 1)))
})
expect(t, "for a == 5 {}", func(p pfn) []ast.Stmt {
return stmts(
forStmt(
nil,
binaryExpr(
ident("a", p(1, 5)),
intLit(5, p(1, 10)),
token.Equal,
p(1, 7)),
nil,
blockStmt(p(1, 12), p(1, 13)),
p(1, 1)))
})
expect(t, "for a := 0; a == 5; {}", func(p pfn) []ast.Stmt {
return stmts(
forStmt(
assignStmt(
exprs(ident("a", p(1, 5))),
exprs(intLit(0, p(1, 10))),
token.Define, p(1, 7)),
binaryExpr(
ident("a", p(1, 13)),
intLit(5, p(1, 18)),
token.Equal,
p(1, 15)),
nil,
blockStmt(p(1, 22), p(1, 23)),
p(1, 1)))
})
expect(t, "for a := 0; a < 5; a++ {}", func(p pfn) []ast.Stmt {
return stmts(
forStmt(
assignStmt(
exprs(ident("a", p(1, 5))),
exprs(intLit(0, p(1, 10))),
token.Define, p(1, 7)),
binaryExpr(
ident("a", p(1, 13)),
intLit(5, p(1, 17)),
token.Less,
p(1, 15)),
incDecStmt(
ident("a", p(1, 20)),
token.Inc, p(1, 21)),
blockStmt(p(1, 24), p(1, 25)),
p(1, 1)))
})
expect(t, "for ; a < 5; a++ {}", func(p pfn) []ast.Stmt {
return stmts(
forStmt(
nil,
binaryExpr(
ident("a", p(1, 7)),
intLit(5, p(1, 11)),
token.Less,
p(1, 9)),
incDecStmt(
ident("a", p(1, 14)),
token.Inc, p(1, 15)),
blockStmt(p(1, 18), p(1, 19)),
p(1, 1)))
})
expect(t, "for a := 0; ; a++ {}", func(p pfn) []ast.Stmt {
return stmts(
forStmt(
assignStmt(
exprs(ident("a", p(1, 5))),
exprs(intLit(0, p(1, 10))),
token.Define, p(1, 7)),
nil,
incDecStmt(
ident("a", p(1, 15)),
token.Inc, p(1, 16)),
blockStmt(p(1, 19), p(1, 20)),
p(1, 1)))
})
expect(t, "for a == 5 && b != 4 {}", func(p pfn) []ast.Stmt {
return stmts(
forStmt(
nil,
binaryExpr(
binaryExpr(
ident("a", p(1, 5)),
intLit(5, p(1, 10)),
token.Equal,
p(1, 7)),
binaryExpr(
ident("b", p(1, 15)),
intLit(4, p(1, 20)),
token.NotEqual,
p(1, 17)),
token.LAnd,
p(1, 12)),
nil,
blockStmt(p(1, 22), p(1, 23)),
p(1, 1)))
})
}

86
compiler/parser/parser_function_test.go
View file

@ -1,86 +0,0 @@
package parser_test
import (
"testing"
"github.com/d5/tengo/compiler/ast"
"github.com/d5/tengo/compiler/token"
)
func TestFunction(t *testing.T) {
expect(t, "a = func(b, c, d) { return d }", func(p pfn) []ast.Stmt {
return stmts(
assignStmt(
exprs(
ident("a", p(1, 1))),
exprs(
funcLit(
funcType(
identList(p(1, 9), p(1, 17), false,
ident("b", p(1, 10)),
ident("c", p(1, 13)),
ident("d", p(1, 16))),
p(1, 5)),
blockStmt(p(1, 19), p(1, 30),
returnStmt(p(1, 21), ident("d", p(1, 28)))))),
token.Assign,
p(1, 3)))
})
}
func TestVariableFunction(t *testing.T) {
expect(t, "a = func(...args) { return args }", func(p pfn) []ast.Stmt {
return stmts(
assignStmt(
exprs(
ident("a", p(1, 1))),
exprs(
funcLit(
funcType(
identList(
p(1, 9), p(1, 17),
true,
ident("args", p(1, 13)),
), p(1, 5)),
blockStmt(p(1, 19), p(1, 33),
returnStmt(p(1, 21),
ident("args", p(1, 28)),
),
),
),
),
token.Assign,
p(1, 3)))
})
}
func TestVariableFunctionWithArgs(t *testing.T) {
expect(t, "a = func(x, y, ...z) { return z }", func(p pfn) []ast.Stmt {
return stmts(
assignStmt(
exprs(
ident("a", p(1, 1))),
exprs(
funcLit(
funcType(
identList(
p(1, 9), p(1, 20),
true,
ident("x", p(1, 10)),
ident("y", p(1, 13)),
ident("z", p(1, 19)),
), p(1, 5)),
blockStmt(p(1, 22), p(1, 33),
returnStmt(p(1, 24),
ident("z", p(1, 31)),
),
),
),
),
token.Assign,
p(1, 3)))
})
expectError(t, "a = func(x, y, ...z, invalid) { return z }")
expectError(t, "a = func(...args, invalid) { return args }")
}

215
compiler/parser/parser_if_test.go
View file

@ -1,215 +0,0 @@
package parser_test
import (
"testing"
"github.com/d5/tengo/compiler/ast"
"github.com/d5/tengo/compiler/token"
)
func TestIf(t *testing.T) {
expect(t, "if a == 5 {}", func(p pfn) []ast.Stmt {
return stmts(
ifStmt(
nil,
binaryExpr(
ident("a", p(1, 4)),
intLit(5, p(1, 9)),
token.Equal,
p(1, 6)),
blockStmt(
p(1, 11), p(1, 12)),
nil,
p(1, 1)))
})
expect(t, "if a == 5 && b != 3 {}", func(p pfn) []ast.Stmt {
return stmts(
ifStmt(
nil,
binaryExpr(
binaryExpr(
ident("a", p(1, 4)),
intLit(5, p(1, 9)),
token.Equal,
p(1, 6)),
binaryExpr(
ident("b", p(1, 14)),
intLit(3, p(1, 19)),
token.NotEqual,
p(1, 16)),
token.LAnd,
p(1, 11)),
blockStmt(
p(1, 21), p(1, 22)),
nil,
p(1, 1)))
})
expect(t, "if a == 5 { a = 3; a = 1 }", func(p pfn) []ast.Stmt {
return stmts(
ifStmt(
nil,
binaryExpr(
ident("a", p(1, 4)),
intLit(5, p(1, 9)),
token.Equal,
p(1, 6)),
blockStmt(
p(1, 11), p(1, 26),
assignStmt(
exprs(ident("a", p(1, 13))),
exprs(intLit(3, p(1, 17))),
token.Assign,
p(1, 15)),
assignStmt(
exprs(ident("a", p(1, 20))),
exprs(intLit(1, p(1, 24))),
token.Assign,
p(1, 22))),
nil,
p(1, 1)))
})
expect(t, "if a == 5 { a = 3; a = 1 } else { a = 2; a = 4 }", func(p pfn) []ast.Stmt {
return stmts(
ifStmt(
nil,
binaryExpr(
ident("a", p(1, 4)),
intLit(5, p(1, 9)),
token.Equal,
p(1, 6)),
blockStmt(
p(1, 11), p(1, 26),
assignStmt(
exprs(ident("a", p(1, 13))),
exprs(intLit(3, p(1, 17))),
token.Assign,
p(1, 15)),
assignStmt(
exprs(ident("a", p(1, 20))),
exprs(intLit(1, p(1, 24))),
token.Assign,
p(1, 22))),
blockStmt(
p(1, 33), p(1, 48),
assignStmt(
exprs(ident("a", p(1, 35))),
exprs(intLit(2, p(1, 39))),
token.Assign,
p(1, 37)),
assignStmt(
exprs(ident("a", p(1, 42))),
exprs(intLit(4, p(1, 46))),
token.Assign,
p(1, 44))),
p(1, 1)))
})
expect(t, `
if a == 5 {
b = 3
c = 1
} else if d == 3 {
e = 8
f = 3
} else {
g = 2
h = 4
}`, func(p pfn) []ast.Stmt {
return stmts(
ifStmt(
nil,
binaryExpr(
ident("a", p(2, 4)),
intLit(5, p(2, 9)),
token.Equal,
p(2, 6)),
blockStmt(
p(2, 11), p(5, 1),
assignStmt(
exprs(ident("b", p(3, 2))),
exprs(intLit(3, p(3, 6))),
token.Assign,
p(3, 4)),
assignStmt(
exprs(ident("c", p(4, 2))),
exprs(intLit(1, p(4, 6))),
token.Assign,
p(4, 4))),
ifStmt(
nil,
binaryExpr(
ident("d", p(5, 11)),
intLit(3, p(5, 16)),
token.Equal,
p(5, 13)),
blockStmt(
p(5, 18), p(8, 1),
assignStmt(
exprs(ident("e", p(6, 2))),
exprs(intLit(8, p(6, 6))),
token.Assign,
p(6, 4)),
assignStmt(
exprs(ident("f", p(7, 2))),
exprs(intLit(3, p(7, 6))),
token.Assign,
p(7, 4))),
blockStmt(
p(8, 8), p(11, 1),
assignStmt(
exprs(ident("g", p(9, 2))),
exprs(intLit(2, p(9, 6))),
token.Assign,
p(9, 4)),
assignStmt(
exprs(ident("h", p(10, 2))),
exprs(intLit(4, p(10, 6))),
token.Assign,
p(10, 4))),
p(5, 8)),
p(2, 1)))
})
expect(t, "if a := 3; a < b {}", func(p pfn) []ast.Stmt {
return stmts(
ifStmt(
assignStmt(
exprs(ident("a", p(1, 4))),
exprs(intLit(3, p(1, 9))),
token.Define, p(1, 6)),
binaryExpr(
ident("a", p(1, 12)),
ident("b", p(1, 16)),
token.Less, p(1, 14)),
blockStmt(
p(1, 18), p(1, 19)),
nil,
p(1, 1)))
})
expect(t, "if a++; a < b {}", func(p pfn) []ast.Stmt {
return stmts(
ifStmt(
incDecStmt(ident("a", p(1, 4)), token.Inc, p(1, 5)),
binaryExpr(
ident("a", p(1, 9)),
ident("b", p(1, 13)),
token.Less, p(1, 11)),
blockStmt(
p(1, 15), p(1, 16)),
nil,
p(1, 1)))
})
expectError(t, `if {}`)
expectError(t, `if a == b { } else a != b { }`)
expectError(t, `if a == b { } else if { }`)
expectError(t, `else { }`)
expectError(t, `if ; {}`)
expectError(t, `if a := 3; {}`)
expectError(t, `if ; a < 3 {}`)
}

46
compiler/parser/parser_import_test.go
View file

@ -1,46 +0,0 @@
package parser_test
import (
"testing"
"github.com/d5/tengo/compiler/ast"
"github.com/d5/tengo/compiler/token"
)
func TestImport(t *testing.T) {
expect(t, `a := import("mod1")`, func(p pfn) []ast.Stmt {
return stmts(
assignStmt(
exprs(ident("a", p(1, 1))),
exprs(importExpr("mod1", p(1, 6))),
token.Define, p(1, 3)))
})
expect(t, `import("mod1").var1`, func(p pfn) []ast.Stmt {
return stmts(
exprStmt(
selectorExpr(
importExpr("mod1", p(1, 1)),
stringLit("var1", p(1, 16)))))
})
expect(t, `import("mod1").func1()`, func(p pfn) []ast.Stmt {
return stmts(
exprStmt(
callExpr(
selectorExpr(
importExpr("mod1", p(1, 1)),
stringLit("func1", p(1, 16))),
p(1, 21), p(1, 22))))
})
expect(t, `for x, y in import("mod1") {}`, func(p pfn) []ast.Stmt {
return stmts(
forInStmt(
ident("x", p(1, 5)),
ident("y", p(1, 8)),
importExpr("mod1", p(1, 13)),
blockStmt(p(1, 28), p(1, 29)),
p(1, 1)))
})
}

98
compiler/parser/parser_index_test.go
View file

@ -1,98 +0,0 @@
package parser_test
import (
"testing"
"github.com/d5/tengo/compiler/ast"
"github.com/d5/tengo/compiler/token"
)
func TestIndex(t *testing.T) {
expect(t, "[1, 2, 3][1]", func(p pfn) []ast.Stmt {
return stmts(
exprStmt(
indexExpr(
arrayLit(p(1, 1), p(1, 9),
intLit(1, p(1, 2)),
intLit(2, p(1, 5)),
intLit(3, p(1, 8))),
intLit(1, p(1, 11)),
p(1, 10), p(1, 12))))
})
expect(t, "[1, 2, 3][5 - a]", func(p pfn) []ast.Stmt {
return stmts(
exprStmt(
indexExpr(
arrayLit(p(1, 1), p(1, 9),
intLit(1, p(1, 2)),
intLit(2, p(1, 5)),
intLit(3, p(1, 8))),
binaryExpr(
intLit(5, p(1, 11)),
ident("a", p(1, 15)),
token.Sub,
p(1, 13)),
p(1, 10), p(1, 16))))
})
expect(t, "[1, 2, 3][5 : a]", func(p pfn) []ast.Stmt {
return stmts(
exprStmt(
sliceExpr(
arrayLit(p(1, 1), p(1, 9),
intLit(1, p(1, 2)),
intLit(2, p(1, 5)),
intLit(3, p(1, 8))),
intLit(5, p(1, 11)),
ident("a", p(1, 15)),
p(1, 10), p(1, 16))))
})
expect(t, "[1, 2, 3][a + 3 : b - 8]", func(p pfn) []ast.Stmt {
return stmts(
exprStmt(
sliceExpr(
arrayLit(p(1, 1), p(1, 9),
intLit(1, p(1, 2)),
intLit(2, p(1, 5)),
intLit(3, p(1, 8))),
binaryExpr(
ident("a", p(1, 11)),
intLit(3, p(1, 15)),
token.Add,
p(1, 13)),
binaryExpr(
ident("b", p(1, 19)),
intLit(8, p(1, 23)),
token.Sub,
p(1, 21)),
p(1, 10), p(1, 24))))
})
expect(t, `{a: 1, b: 2}["b"]`, func(p pfn) []ast.Stmt {
return stmts(
exprStmt(
indexExpr(
mapLit(p(1, 1), p(1, 12),
mapElementLit("a", p(1, 2), p(1, 3), intLit(1, p(1, 5))),
mapElementLit("b", p(1, 8), p(1, 9), intLit(2, p(1, 11)))),
stringLit("b", p(1, 14)),
p(1, 13), p(1, 17))))
})
expect(t, `{a: 1, b: 2}[a + b]`, func(p pfn) []ast.Stmt {
return stmts(
exprStmt(
indexExpr(
mapLit(p(1, 1), p(1, 12),
mapElementLit("a", p(1, 2), p(1, 3), intLit(1, p(1, 5))),
mapElementLit("b", p(1, 8), p(1, 9), intLit(2, p(1, 11)))),
binaryExpr(
ident("a", p(1, 14)),
ident("b", p(1, 18)),
token.Add,
p(1, 16)),
p(1, 13), p(1, 19))))
})
}

54
compiler/parser/parser_logical_test.go
View file

@ -1,54 +0,0 @@
package parser_test
import (
"testing"
"github.com/d5/tengo/compiler/ast"
"github.com/d5/tengo/compiler/token"
)
func TestLogical(t *testing.T) {
expect(t, "a && 5 || true", func(p pfn) []ast.Stmt {
return stmts(
exprStmt(
binaryExpr(
binaryExpr(
ident("a", p(1, 1)),
intLit(5, p(1, 6)),
token.LAnd,
p(1, 3)),
boolLit(true, p(1, 11)),
token.LOr,
p(1, 8))))
})
expect(t, "a || 5 && true", func(p pfn) []ast.Stmt {
return stmts(
exprStmt(
binaryExpr(
ident("a", p(1, 1)),
binaryExpr(
intLit(5, p(1, 6)),
boolLit(true, p(1, 11)),
token.LAnd,
p(1, 8)),
token.LOr,
p(1, 3))))
})
expect(t, "a && (5 || true)", func(p pfn) []ast.Stmt {
return stmts(
exprStmt(
binaryExpr(
ident("a", p(1, 1)),
parenExpr(
binaryExpr(
intLit(5, p(1, 7)),
boolLit(true, p(1, 12)),
token.LOr,
p(1, 9)),
p(1, 6), p(1, 16)),
token.LAnd,
p(1, 3))))
})
}

80
compiler/parser/parser_map_test.go
View file

@ -1,80 +0,0 @@
package parser_test
import (
"testing"
"github.com/d5/tengo/compiler/ast"
"github.com/d5/tengo/compiler/token"
)
func TestMap(t *testing.T) {
expect(t, "{ key1: 1, key2: \"2\", key3: true }", func(p pfn) []ast.Stmt {
return stmts(
exprStmt(
mapLit(p(1, 1), p(1, 34),
mapElementLit("key1", p(1, 3), p(1, 7), intLit(1, p(1, 9))),
mapElementLit("key2", p(1, 12), p(1, 16), stringLit("2", p(1, 18))),
mapElementLit("key3", p(1, 23), p(1, 27), boolLit(true, p(1, 29))))))
})
expect(t, "{ \"key1\": 1 }", func(p pfn) []ast.Stmt {
return stmts(
exprStmt(
mapLit(p(1, 1), p(1, 13),
mapElementLit("key1", p(1, 3), p(1, 9), intLit(1, p(1, 11))))))
})
expect(t, "a = { key1: 1, key2: \"2\", key3: true }", func(p pfn) []ast.Stmt {
return stmts(
assignStmt(
exprs(ident("a", p(1, 1))),
exprs(mapLit(p(1, 5), p(1, 38),
mapElementLit("key1", p(1, 7), p(1, 11), intLit(1, p(1, 13))),
mapElementLit("key2", p(1, 16), p(1, 20), stringLit("2", p(1, 22))),
mapElementLit("key3", p(1, 27), p(1, 31), boolLit(true, p(1, 33))))),
token.Assign,
p(1, 3)))
})
expect(t, "a = { key1: 1, key2: \"2\", key3: { k1: `bar`, k2: 4 } }", func(p pfn) []ast.Stmt {
return stmts(
assignStmt(
exprs(ident("a", p(1, 1))),
exprs(mapLit(p(1, 5), p(1, 54),
mapElementLit("key1", p(1, 7), p(1, 11), intLit(1, p(1, 13))),
mapElementLit("key2", p(1, 16), p(1, 20), stringLit("2", p(1, 22))),
mapElementLit("key3", p(1, 27), p(1, 31),
mapLit(p(1, 33), p(1, 52),
mapElementLit("k1", p(1, 35), p(1, 37), stringLit("bar", p(1, 39))),
mapElementLit("k2", p(1, 46), p(1, 48), intLit(4, p(1, 50))))))),
token.Assign,
p(1, 3)))
})
expect(t, `
{
key1: 1,
key2: "2",
key3: true
}`, func(p pfn) []ast.Stmt {
return stmts(
exprStmt(
mapLit(p(2, 1), p(6, 1),
mapElementLit("key1", p(3, 2), p(3, 6), intLit(1, p(3, 8))),
mapElementLit("key2", p(4, 2), p(4, 6), stringLit("2", p(4, 8))),
mapElementLit("key3", p(5, 2), p(5, 6), boolLit(true, p(5, 8))))))
})
expectError(t, `
{
key1: 1,
key2: "2",
key3: true,
}`) // unlike Go, trailing comma for the last element is illegal
expectError(t, `{ key1: 1, }`)
expectError(t, `{
key1: 1,
key2: 2,
}`)
}

11
compiler/parser/parser_precendence_test.go
View file

@ -1,11 +0,0 @@
package parser_test
import (
"testing"
)
func TestPrecedence(t *testing.T) {
expectString(t, `a + b + c`, `((a + b) + c)`)
expectString(t, `a + b * c`, `(a + (b * c))`)
expectString(t, `x = 2 * 1 + 3 / 4`, `x = ((2 * 1) + (3 / 4))`)
}

165
compiler/parser/parser_selector_test.go
View file

@ -1,165 +0,0 @@
package parser_test
import (
"testing"
"github.com/d5/tengo/compiler/ast"
"github.com/d5/tengo/compiler/token"
)
func TestSelector(t *testing.T) {
expect(t, "a.b", func(p pfn) []ast.Stmt {
return stmts(
exprStmt(
selectorExpr(
ident("a", p(1, 1)),
stringLit("b", p(1, 3)))))
})
expect(t, "a.b.c", func(p pfn) []ast.Stmt {
return stmts(
exprStmt(
selectorExpr(
selectorExpr(
ident("a", p(1, 1)),
stringLit("b", p(1, 3))),
stringLit("c", p(1, 5)))))
})
expect(t, "{k1:1}.k1", func(p pfn) []ast.Stmt {
return stmts(
exprStmt(
selectorExpr(
mapLit(
p(1, 1), p(1, 6),
mapElementLit("k1", p(1, 2), p(1, 4), intLit(1, p(1, 5)))),
stringLit("k1", p(1, 8)))))
})
expect(t, "{k1:{v1:1}}.k1.v1", func(p pfn) []ast.Stmt {
return stmts(
exprStmt(
selectorExpr(
selectorExpr(
mapLit(
p(1, 1), p(1, 11),
mapElementLit("k1", p(1, 2), p(1, 4),
mapLit(p(1, 5), p(1, 10),
mapElementLit("v1", p(1, 6), p(1, 8), intLit(1, p(1, 9)))))),
stringLit("k1", p(1, 13))),
stringLit("v1", p(1, 16)))))
})
expect(t, "a.b = 4", func(p pfn) []ast.Stmt {
return stmts(
assignStmt(
exprs(selectorExpr(ident("a", p(1, 1)), stringLit("b", p(1, 3)))),
exprs(intLit(4, p(1, 7))),
token.Assign, p(1, 5)))
})
expect(t, "a.b.c = 4", func(p pfn) []ast.Stmt {
return stmts(
assignStmt(
exprs(selectorExpr(selectorExpr(ident("a", p(1, 1)), stringLit("b", p(1, 3))), stringLit("c", p(1, 5)))),
exprs(intLit(4, p(1, 9))),
token.Assign, p(1, 7)))
})
expect(t, "a.b.c = 4 + 5", func(p pfn) []ast.Stmt {
return stmts(
assignStmt(
exprs(selectorExpr(selectorExpr(ident("a", p(1, 1)), stringLit("b", p(1, 3))), stringLit("c", p(1, 5)))),
exprs(binaryExpr(intLit(4, p(1, 9)), intLit(5, p(1, 13)), token.Add, p(1, 11))),
token.Assign, p(1, 7)))
})
expect(t, "a[0].c = 4", func(p pfn) []ast.Stmt {
return stmts(
assignStmt(
exprs(
selectorExpr(
indexExpr(
ident("a", p(1, 1)),
intLit(0, p(1, 3)),
p(1, 2), p(1, 4)),
stringLit("c", p(1, 6)))),
exprs(intLit(4, p(1, 10))),
token.Assign, p(1, 8)))
})
expect(t, "a.b[0].c = 4", func(p pfn) []ast.Stmt {
return stmts(
assignStmt(
exprs(
selectorExpr(
indexExpr(
selectorExpr(
ident("a", p(1, 1)),
stringLit("b", p(1, 3))),
intLit(0, p(1, 5)),
p(1, 4), p(1, 6)),
stringLit("c", p(1, 8)))),
exprs(intLit(4, p(1, 12))),
token.Assign, p(1, 10)))
})
expect(t, "a.b[0][2].c = 4", func(p pfn) []ast.Stmt {
return stmts(
assignStmt(
exprs(
selectorExpr(
indexExpr(
indexExpr(
selectorExpr(
ident("a", p(1, 1)),
stringLit("b", p(1, 3))),
intLit(0, p(1, 5)),
p(1, 4), p(1, 6)),
intLit(2, p(1, 8)),
p(1, 7), p(1, 9)),
stringLit("c", p(1, 11)))),
exprs(intLit(4, p(1, 15))),
token.Assign, p(1, 13)))
})
expect(t, `a.b["key1"][2].c = 4`, func(p pfn) []ast.Stmt {
return stmts(
assignStmt(
exprs(
selectorExpr(
indexExpr(
indexExpr(
selectorExpr(
ident("a", p(1, 1)),
stringLit("b", p(1, 3))),
stringLit("key1", p(1, 5)),
p(1, 4), p(1, 11)),
intLit(2, p(1, 13)),
p(1, 12), p(1, 14)),
stringLit("c", p(1, 16)))),
exprs(intLit(4, p(1, 20))),
token.Assign, p(1, 18)))
})
expect(t, "a[0].b[2].c = 4", func(p pfn) []ast.Stmt {
return stmts(
assignStmt(
exprs(
selectorExpr(
indexExpr(
selectorExpr(
indexExpr(
ident("a", p(1, 1)),
intLit(0, p(1, 3)),
p(1, 2), p(1, 4)),
stringLit("b", p(1, 6))),
intLit(2, p(1, 8)),
p(1, 7), p(1, 9)),
stringLit("c", p(1, 11)))),
exprs(intLit(4, p(1, 15))),
token.Assign, p(1, 13)))
})
expectError(t, `a.(b.c)`)
}

Some files were not shown because too many files have changed in this diff Show more